crunchdao / crunch-cli

7 stars 4 forks source link

Monkey Patch: `pytorch_lightning`'s logging #155

Open Caceresenzo opened 4 weeks ago

Caceresenzo commented 4 weeks ago

https://hub.crunchdao.com/competitions/causality-discovery/projects/5130/first-submission/runs/23455

Traceback (most recent call last):
  File "/context/venv/lib/python3.10/site-packages/crunch/runner/cloud_executor.py", line 229, in start
    prediction = self.process_linear(
  File "/context/venv/lib/python3.10/site-packages/crunch/runner/cloud_executor.py", line 295, in process_linear
    utils.smart_call(
  File "/context/venv/lib/python3.10/site-packages/crunch/utils.py", line 260, in smart_call
    return function(**arguments)
  File "/context/code/main.py", line 1398, in train
    trainer.fit(model, train_loader)
  File "/context/venv/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 538, in fit
    call._call_and_handle_interrupt(
  File "/context/venv/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 47, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/context/venv/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 574, in _fit_impl
    self._run(model, ckpt_path=ckpt_path)
  File "/context/venv/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 943, in _run
    call._call_setup_hook(self)  # allow user to set up LightningModule in accelerator environment
  File "/context/venv/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 96, in _call_setup_hook
    if hasattr(logger, "experiment"):
  File "/context/venv/lib/python3.10/site-packages/lightning_fabric/loggers/logger.py", line 118, in experiment
    return fn(self)
  File "/context/venv/lib/python3.10/site-packages/pytorch_lightning/loggers/csv_logs.py", line 166, in experiment
    self._fs.makedirs(self.root_dir, exist_ok=True)
  File "/context/venv/lib/python3.10/site-packages/fsspec/implementations/local.py", line 54, in makedirs
    os.makedirs(path, exist_ok=exist_ok)
  File "/usr/local/lib/python3.10/os.py", line 225, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/context/code/lightning_logs'