Erotemic
commented
1 year ago @cjrd @RitwikGupta I'm trying to get a MWE of this running. With the latest changes you can do something like this:
# Create demo train / vali data
DATA_PATH=$(python -m scalemae.demo)
echo "
data:
type: ImageList
length: 10
img_dir: '$DATA_PATH'
mean: [0.46921533, 0.46026663, 0.41329921]
std: [0.1927, 0.1373, 0.1203]
vis_factor: 1.0
" > $DATA_PATH/demo.yaml
cat $DATA_PATH/demo.yaml
DEFAULT_ROOT_DIR=$HOME/exps/scalemae_demo
echo "
DEFAULT_ROOT_DIR = $DEFAULT_ROOT_DIR
DATA_PATH = $DATA_PATH
"
mkdir -p $DEFAULT_ROOT_DIR
CUDA_VISIBLE_DEVICES=0,1 python -m torch.distributed.launch --nproc_per_node=1 --master_port=11085 -m scalemae.main_pretrain \
--output_dir $DEFAULT_ROOT_DIR \
--log_dir $DEFAULT_ROOT_DIR \
--config $DATA_PATH/demo.yaml \
--eval_path "$DATA_PATH" \
--batch_size 4 \
--model mae_vit_base_patch16 \
--mask_ratio 0.75 \
--num_workers 0 \
--epochs 300 \
--target_size 224\
--input_size 224\
--self_attention\
--scale_min 0.2 \
--scale_max 1.0 \
--warmup_epochs 40 \
--blr 1.5e-4 --weight_decay 0.05 \
--decoder_aux_loss_layers 1\
--target_size_scheduler constant\
--decoder_depth 8 \
--no_autoresume \
--use_mask_token \
--skip_knn_eval \
--fixed_output_size_min 224\
--fixed_output_size_max 336\
--absolute_scale This generates a small dataset with kwcoco, so it can grow larger if needed. I'm able to write an ImageFolder that should corresond to one of the dataloaders. I thought the above would run, but I got:
RuntimeError: Unexpected error from cudaGetDeviceCount().
```
data:
type: ImageList
length: 10
img_dir: '/home/joncrall/.cache/scalemae/tests/demo/imagefolder'
mean: [0.46921533, 0.46026663, 0.41329921]
std: [0.1927, 0.1373, 0.1203]
vis_factor: 1.0
DEFAULT_ROOT_DIR = /home/joncrall/exps/scalemae_demo
DATA_PATH = /home/joncrall/.cache/scalemae/tests/demo/imagefolder
/home/joncrall/.pyenv/versions/3.11.2/envs/pyenv3.11.2/lib/python3.11/site-packages/torch/distributed/launch.py:181: FutureWarning: The module torch.distributed.launch is deprecated
and will be removed in future. Use torchrun.
Note that --use-env is set by default in torchrun.
If your script expects `--local-rank` argument to be set, please
change it to read from `os.environ['LOCAL_RANK']` instead. See
https://pytorch.org/docs/stable/distributed.html#launch-utility for
further instructions
warnings.warn(
2023-11-04 21:27:40.159162: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-11-04 21:27:40.201501: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-11-04 21:27:41.136947: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
Starting pretrain
Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "/home/joncrall/code/watch/geowatch_tpl/submodules/scale-mae/scalemae/main_pretrain.py", line 771, in
main(args)
File "/home/joncrall/.pyenv/versions/3.11.2/envs/pyenv3.11.2/lib/python3.11/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
return f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/home/joncrall/code/watch/geowatch_tpl/submodules/scale-mae/scalemae/main_pretrain.py", line 409, in main
misc.init_distributed_mode(args)
File "/home/joncrall/code/watch/geowatch_tpl/submodules/scale-mae/scalemae/util/misc.py", line 264, in init_distributed_mode
torch.cuda.set_device(args.gpu)
File "/home/joncrall/.pyenv/versions/3.11.2/envs/pyenv3.11.2/lib/python3.11/site-packages/torch/cuda/__init__.py", line 350, in set_device
torch._C._cuda_setDevice(device)
File "/home/joncrall/.pyenv/versions/3.11.2/envs/pyenv3.11.2/lib/python3.11/site-packages/torch/cuda/__init__.py", line 247, in _lazy_init
torch._C._cuda_init()
RuntimeError: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 804: forward compatibility was attempted on non supported HW
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 518179) of binary: /home/joncrall/.pyenv/versions/3.11.2/envs/pyenv3.11.2/bin/python
Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "/home/joncrall/.pyenv/versions/3.11.2/envs/pyenv3.11.2/lib/python3.11/site-packages/torch/distributed/launch.py", line 196, in
main()
File "/home/joncrall/.pyenv/versions/3.11.2/envs/pyenv3.11.2/lib/python3.11/site-packages/torch/distributed/launch.py", line 192, in main
launch(args)
File "/home/joncrall/.pyenv/versions/3.11.2/envs/pyenv3.11.2/lib/python3.11/site-packages/torch/distributed/launch.py", line 177, in launch
run(args)
File "/home/joncrall/.pyenv/versions/3.11.2/envs/pyenv3.11.2/lib/python3.11/site-packages/torch/distributed/run.py", line 785, in run
elastic_launch(
File "/home/joncrall/.pyenv/versions/3.11.2/envs/pyenv3.11.2/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
return launch_agent(self._config, self._entrypoint, list(args))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/joncrall/.pyenv/versions/3.11.2/envs/pyenv3.11.2/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
scalemae.main_pretrain FAILED
------------------------------------------------------------
Failures:
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
time : 2023-11-04_21:27:41
host : toothbrush
rank : 0 (local_rank: 0)
exitcode : 1 (pid: 518179)
error_file: /tmp/torchelastic_ktnaatzu/none_6fhs8u23/attempt_0/0/error.json
traceback : Traceback (most recent call last):
File "/home/joncrall/.pyenv/versions/3.11.2/envs/pyenv3.11.2/lib/python3.11/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
return f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/home/joncrall/code/watch/geowatch_tpl/submodules/scale-mae/scalemae/main_pretrain.py", line 409, in main
misc.init_distributed_mode(args)
File "/home/joncrall/code/watch/geowatch_tpl/submodules/scale-mae/scalemae/util/misc.py", line 264, in init_distributed_mode
torch.cuda.set_device(args.gpu)
File "/home/joncrall/.pyenv/versions/3.11.2/envs/pyenv3.11.2/lib/python3.11/site-packages/torch/cuda/__init__.py", line 350, in set_device
torch._C._cuda_setDevice(device)
File "/home/joncrall/.pyenv/versions/3.11.2/envs/pyenv3.11.2/lib/python3.11/site-packages/torch/cuda/__init__.py", line 247, in _lazy_init
torch._C._cuda_init()
RuntimeError: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 804: forward compatibility was attempted on non supported HW
============================================================
```
This could just be a hardware problem (can this not run on 2x 3090's?). Is there anything obviously wrong about my config?
Are there recommended settings for attempting to reproduce the pipeline on a small dataset (for testing).
RitwikGupta
I'm looking into integrating ScaleMAE into geowatch. I've made this branch to track modifications to make it work. Currently this involves:
Setting up proper package namespaces: Everything should be referenced under the "scalemae" namespace to allow for integrations with other libraries. Having a module named "lib" is a common anti-pattern in repos, as it leads to conflicts, and simply putting everything into a top-level namespace fixes this issue. It also means all imports are now referenced explicitly in the code itself.
Finding minimum versions of required and optional dependencies. Still working on this, but there doesn't seem to be a comprehensive list of requirements to make the repo work. I'm working on gathering those while also deconflicting with requirements of geowatch.
Linting to remove unused code.
This should not be merged yet. I'm just ensuring the work is pushed as it is developed for comments and visibility.