reloadware / reloadium

Hot Reloading and Profiling for Python
https://reloadium.io
Apache License 2.0
2.74k stars 56 forks source link

Reloadium appears to not support debugging on remote servers. While it works fine during local testing, it fails to update on servers. #168

Closed TalonX1 closed 9 months ago

dkrystki commented 9 months ago

Hi @TalonX1 have you added path mappings? Without path mappings reloadium doesn't know how to resolve the paths so it is necessary.

TalonX1 commented 9 months ago

Are you referring to the remote mapping feature in PyCharm? I have added it, and debugging works fine, but the hot reloading during debugging does not seem to take effect. When I modify the code and save it, I encounter the error message shown in Image 2. It appears to be concatenating the server path and the local Windows path, and I'm not sure where the issue is originating from.

/data2/qhb/anaconda3/envs/mask3d_cuda113/bin/python3 -m reloadium_launcher pydev_proxy /home/qhb/.pycharm_helpers/pydev/pydevd.py --multiprocess --qt-support=auto --client localhost --port 45219 --file /data2/qhb/mask3d/Mask3D-main_20230621/det_debug_main_instance_segmentation.py general.experiment_name=benchmark_04 general.eval_on_segments=true general.train_on_segments=true data.train_mode=train_validation data.num_workers=0 data.batch_size=2 
Connected to pydev debugger (build 231.9225.15)
■■■■■■■■■■■■■■■
Reloadium 1.2.1
■■■■■■■■■■■■■■■
If you like this project consider becoming a sponsor or giving a star at https://github.com/reloadware/reloadium
/data2/qhb/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/MinkowskiEngine-0.5.4-py3.10-linux-x86_64.egg/MinkowskiEngine/__init__.py:36: UserWarning: The environment variable `OMP_NUM_THREADS` not set. MinkowskiEngine will automatically set `OMP_NUM_THREADS=16`. If you want to set `OMP_NUM_THREADS` manually, please export it on the command line before running a python script. e.g. `export OMP_NUM_THREADS=12; python your_program.py`. It is recommended to set it below 24.
  warnings.warn(
Loaded 44 watched modules so far from paths:
 - /data2/qhb/mask3d/Mask3D-main_20230621/**/*.py
/data2/qhb/mask3d/Mask3D-main_20230621
/data2/qhb/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/pytorch_lightning/utilities/seed.py:55: UserWarning: No seed found, seed set to 2100419569
  rank_zero_warn(f"No seed found, seed set to {seed}")
Global seed set to 2100419569
{'_target_': 'pytorch_lightning.loggers.TensorBoardLogger', 'name': '${general.experiment_id}', 'version': '${general.version}', 'save_dir': '${general.save_dir}'}
{'_target_': 'pytorch_lightning.loggers.CSVLogger', 'name': '${general.experiment_id}', 'version': '${general.version}', 'save_dir': '${general.save_dir}'}
/data2/qhb/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/pytorch_lightning/loggers/csv_logs.py:57: UserWarning: Experiment logs directory saved/benchmark_04/det_scannet_debug/version_1 exists and is not empty. Previous log files in this directory will be deleted when the new ones are saved!
  rank_zero_warn(
[2023-09-24 18:46:29,873][__main__][INFO] - {'general_train_mode': True, 'general_task': 'instance_segmentation'}
/data2/qhb/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py:446: LightningDeprecationWarning: Setting `Trainer(gpus=[0])` is deprecated in v1.7 and will be removed in v2.0. Please use `Trainer(accelerator='gpu', devices=[0])` instead.
  rank_zero_deprecation(
/data2/qhb/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/callback_connector.py:57: LightningDeprecationWarning: Setting `Trainer(weights_save_path=)` has been deprecated in v1.6 and will be removed in v1.8. Please pass ``dirpath`` directly to the `ModelCheckpoint` callback
  rank_zero_deprecation(
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
/data2/qhb/mask3d/Mask3D-main_20230621/datasets/det_semseg.py:641: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  file = yaml.load(f)
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [2]
  | Name      | Type            | Params
----------------------------------------------
0 | model     | DetMask3D       | 41.9 M
1 | criterion | SetCriterionDet | 0     
----------------------------------------------
41.9 M    Trainable params
0         Non-trainable params
41.9 M    Total params
167.731   Total estimated model params size (MB)
/data2/qhb/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:219: PossibleUserWarning: The dataloader, train_dataloader, does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` (try 64 which is the number of cpus on this machine) in the `DataLoader` init to improve performance.
  rank_zero_warn(
/data2/qhb/anaconda3/envs/mask3d_cuda113/lib/python3.10/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:219: PossibleUserWarning: The dataloader, val_dataloader 0, does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` (try 64 which is the number of cpus on this machine) in the `DataLoader` init to improve performance.
  rank_zero_warn(
Epoch 0:   0%|                                         | 0/1069 [00:00<?, ?it/s]File "/data2/qhb/mask3d/Mask3D-main_20230621/C:/Users/TimeAssassin/Desktop/20230510_mask3d/Mask3D-main/Mask3D-main/models/detmask3d.py" has been modified but is not loaded yet.
Hot reloading not needed.

image

image

dkrystki commented 9 months ago

Could you add path mappings in the Run/Debug configuration showed here:

image The Path Mapping is necessary. Otherwise, image

Source: https://github.com/reloadware/reloadium/issues/64#issuecomment-1324581299

TalonX1 commented 9 months ago

It works,thank you for your help!