search

NASA-IMPACT / hls-foundation-os

This repository contains examples of fine-tuning Harmonized Landsat and Sentinel-2 (HLS) Prithvi foundation model.
Apache License 2.0
319 stars 83 forks source link

Memory issues training with multi_temporal_crop_classification.py #48

Closed robmarkcole closed 11 months ago

robmarkcole commented 11 months ago

Ive tried running the fune tuning with multi_temporal_crop_classification.py on both V100 and A10G but get CUDA out of memory. Can you advise if a machine with larger memory is required, or if there is a way to modify the training config but still reproduce the paper conditions? Thanks

robmarkcole commented 11 months ago

Resolved using smaller batch size, which confusingly is not called batch_size but:

    samples_per_gpu=4, # 8,