Experimental image support

pjbull commented 6 months ago

Runs the existing video models on images. This is not the long-term-support version since it is pretty inefficient, but it's a useful test.

Needs:

[x] tests
[x] docs

This can be tested with:

Install in fresh Python 3.9 environment with pip install git+https://github.com/drivendataorg/zamba@exp-image-to-vid
Running commands prefixed with PREDICT_ON_IMAGES=True

For example, see sample logs and output below

> PREDICT_ON_IMAGES=True zamba predict --data-dir test-images/ --output-class-names --overwrite

2024-04-10 12:36:07.789 | INFO     | zamba.models.config:get_filepaths:221 - Getting files in test-images.
2024-04-10 12:36:07.789 | INFO     | zamba.models.config:get_filepaths:244 - Found 6 videos in test-images.
2024-04-10 12:36:07.791 | INFO     | zamba.models.config:check_files_exist_and_load:114 - Checking all 6 filepaths exist. Trying fast file checking...
QUEUEING TASKS | : 100%|████████████████████████████████████████████████████████████████████████████████████████████| 6/6 [00:00<00:00, 7906.32it/s]
PROCESSING TASKS | : 100%|█████████████████████████████████████████████████████████████████████████████████████████| 6/6 [00:00<00:00, 61832.49it/s]
COLLECTING RESULTS | : 100%|██████████████████████████████████████████████████████████████████████████████████████| 6/6 [00:00<00:00, 120410.64it/s]
2024-04-10 12:36:07.802 | INFO     | zamba.models.config:check_files_exist_and_load:151 - Checking that all videos can be loaded. If you're very confident all your videos can be loaded, you can skip this with `skip_load_validation`, but it's not recommended.
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6/6 [00:00<00:00, 46.74it/s]
The following configuration will be used for inference:

    Config file: None
    Data directory: test-images
    Filepath csv: None
    Model: time_distributed
    Checkpoint: None
    Batch size: 2
    Number of workers: 3
    GPUs: 0
    Dry run: False
    Save directory: /Users/bull/code/zamba
    Proba threshold: None
    Output class names: True
    Weight download region: us

Is this correct? [Y/n]: y
2024-04-10 12:36:12.070 | INFO     | zamba.models.model_manager:instantiate_model:77 - Instantiating model: TimeDistributedEfficientNet
2024-04-10 12:36:12.070 | INFO     | zamba.models.model_manager:instantiate_model:81 - Loading from checkpoint.
Lightning automatically upgraded your loaded checkpoint from v1.7.5 to v2.0.9.post0. To apply the upgrade to your files permanently, run `python -m pytorch_lightning.utilities.upgrade_checkpoint --file ../../Library/Caches/zamba/time_distributed_1d483fc723.ckpt`
2024-04-10 12:36:13.394 | INFO     | zamba.models.model_manager:predict_model:385 - No cache dir is specified. Videos will not be cached.
GPU available: True (mps), used: False
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
/Users/bull/miniconda3/envs/zamba/lib/python3.9/site-packages/pytorch_lightning/trainer/setup.py:201: UserWarning: MPS available but not used. Set `accelerator` and `devices` using `Trainer(accelerator='mps', devices=1)`.
  rank_zero_warn(
2024-04-10 12:36:13.476 | INFO     | zamba.models.model_manager:predict_model:408 - Writing out full configuration to /Users/bull/code/zamba/predict_configuration.yaml.
2024-04-10 12:36:13.478 | INFO     | zamba.models.model_manager:predict_model:413 - Starting prediction...
Predicting: 0it [00:00, ?it/s]/Users/bull/code/zamba/zamba/pytorch/dataloaders.py:148: FutureWarning: Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`
  target = torch.tensor(target).float()
/Users/bull/code/zamba/zamba/pytorch/dataloaders.py:148: FutureWarning: Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`
  target = torch.tensor(target).float()
/Users/bull/code/zamba/zamba/pytorch/dataloaders.py:148: FutureWarning: Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`
  target = torch.tensor(target).float()
/Users/bull/code/zamba/zamba/pytorch/dataloaders.py:148: FutureWarning: Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`
  target = torch.tensor(target).float()
/Users/bull/code/zamba/zamba/pytorch/dataloaders.py:148: FutureWarning: Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`
  target = torch.tensor(target).float()
/Users/bull/code/zamba/zamba/pytorch/dataloaders.py:148: FutureWarning: Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`
  target = torch.tensor(target).float()
Predicting DataLoader 0: 100%|████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:34<00:00, 11.50s/it]
2024-04-10 12:36:50.918 | INFO     | zamba.models.model_manager:predict_model:432 - Saving out predictions to /Users/bull/code/zamba/zamba_predictions.csv.

Output:

cat zamba_predictions.csv
filepath,0
/Users/bull/code/zamba/test-images/chimps.jpeg,chimpanzee_bonobo
/Users/bull/code/zamba/test-images/wild-dog.jpg,wild_dog_jackal
/Users/bull/code/zamba/test-images/elephant.jpg,elephant
/Users/bull/code/zamba/test-images/aardvark.jpg,badger
/Users/bull/code/zamba/test-images/serval.jpg,small_cat
/Users/bull/code/zamba/test-images/zebra.webp,equid

netlify[bot] commented 6 months ago

Deploy Preview for silly-keller-664934 ready!

Name	Link
Latest commit	249ab3c22ec06e8f9c477761b53129215451a258
Latest deploy log	https://app.netlify.com/sites/silly-keller-664934/deploys/661b1751b778f7000892c975
Deploy Preview	https://deploy-preview-314--silly-keller-664934.netlify.app
Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

github-actions[bot] commented 6 months ago

🚀 Deployed on https://deploy-preview-314--silly-keller-664934.netlify.app

codecov[bot] commented 6 months ago

Codecov Report

Attention: Patch coverage is 60.00000% with 6 lines in your changes are missing coverage. Please review.

Project coverage is 87.4%. Comparing base (f11fa5a) to head (249ab3c).

Additional details and impacted files

```diff @@ Coverage Diff @@ ## master #314 +/- ## ======================================== - Coverage 87.6% 87.4% -0.3% ======================================== Files 26 26 Lines 2178 2191 +13 ======================================== + Hits 1908 1915 +7 - Misses 270 276 +6 ``` | [Files](https://app.codecov.io/gh/drivendataorg/zamba/pull/314?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=drivendataorg) | Coverage Δ | | |---|---|---| | [zamba/models/config.py](https://app.codecov.io/gh/drivendataorg/zamba/pull/314?src=pr&el=tree&filepath=zamba%2Fmodels%2Fconfig.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=drivendataorg#diff-emFtYmEvbW9kZWxzL2NvbmZpZy5weQ==) | `97.5% <100.0%> (+<0.1%)` | :arrow_up: | | [zamba/settings.py](https://app.codecov.io/gh/drivendataorg/zamba/pull/314?src=pr&el=tree&filepath=zamba%2Fsettings.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=drivendataorg#diff-emFtYmEvc2V0dGluZ3MucHk=) | `87.5% <100.0%> (+4.1%)` | :arrow_up: | | [zamba/data/video.py](https://app.codecov.io/gh/drivendataorg/zamba/pull/314?src=pr&el=tree&filepath=zamba%2Fdata%2Fvideo.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=drivendataorg#diff-emFtYmEvZGF0YS92aWRlby5weQ==) | `80.8% <33.3%> (-1.7%)` | :arrow_down: |

pjbull commented 6 months ago

Ready for review. Added docs here: https://deploy-preview-314--silly-keller-664934.netlify.app/predict-tutorial/#predicting-species-from-images

drivendataorg / zamba