Error defining `TraceModel` class

[wbenoit26]

When trying to run training, I get an error that says AttributeError: module 'ray.train' has no attribute 'lightning'. I re-created the issue in the snippet below. A solution would be to do from ray.train.lightning import RayTrainReportCallback, but that shouldn't be necessary.

Python 3.10.12 (main, Aug 16 2023, 05:44:31) [GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ray
>>> class Test(ray.train.lightning.RayTrainReportCallback):
...     pass
...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.10/site-packages/ray/__init__.py", line 289, in __getattr__
    raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
AttributeError: module 'ray' has no attribute 'train'
>>> import ray.train
>>> class Test(ray.train.lightning.RayTrainReportCallback):
...     pass
...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'ray.train' has no attribute 'lightning'
>>> import ray.train.lightning
>>> class Test(ray.train.lightning.RayTrainReportCallback):
...     pass
...
>>>

[EthanMarx]

https://github.com/ML4GW/aframev2/blob/main/projects/train/train/callbacks.py

Should have fixed this last week on main, where is it coming from?

[wbenoit26]

Weird, my local repo is up to date and I re-built the container yesterday. Let me double-check everything and see if the issue persists.

[wbenoit26]

Yeah, it looks like importing ray.train doesn't import lightning: https://github.com/ray-project/ray/blob/master/python/ray/train/init.py

[EthanMarx]

Weird, my local repo is up to date and I re-built the container yesterday. Let me double-check everything and see if the issue persists.

Yeah try rebuilding / repulling and if it doesn't work again post the full traceback

[wbenoit26]

❯ poetry run train --config config.yaml
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/william.benoit/miniconda3/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/william.benoit/ML4GW/aframev2/projects/train/train/cli.py", line 7, in <module>
    from train.model import AframeBase
  File "/home/william.benoit/ML4GW/aframev2/projects/train/train/model/__init__.py", line 1, in <module>
    from .autoencoder import AutoencoderAframe
  File "/home/william.benoit/ML4GW/aframev2/projects/train/train/model/autoencoder.py", line 5, in <module>
    from train.model.base import AframeBase
  File "/home/william.benoit/ML4GW/aframev2/projects/train/train/model/base.py", line 9, in <module>
    from train.callbacks import ModelCheckpoint, SaveAugmentedBatch
  File "/home/william.benoit/ML4GW/aframev2/projects/train/train/callbacks.py", line 125, in <module>
    class TraceModel(ray.train.lightning.RayTrainReportCallback):
AttributeError: module 'ray.train' has no attribute 'lightning'

This is after pulling/re-building

[wbenoit26]

But based on that __init__ file, I think explicitly importing ray.train.lightning is required, so I'm curious how it works for you

[wbenoit26]

Oh wait but specifying the full path should work. That's strange.