[Bug] Eval fails with Permission denied: '/run/user' when running from within a devcontainer

[alex-treebeard]

I have patched this locally with a variable "INSPECT_RUNTIME_PATH": "${workspaceFolder}/runtime"

I believe you can repro this by running any eval within a devcontainer

examples/agents/intercode-ctf on  sandbox [$!] via 🐍 v3.11.4 @172.17.0.2 
❯ inspect eval task.py --model mistral/mistral-large-latest -T shuffle=true --limit 1
[+] Building 0.8s (8/8) FINISHED                                                         docker:default
 => [default internal] load build definition from Dockerfile                                       0.1s
 => => transferring dockerfile: 615B                                                               0.0s
 => [default internal] load .dockerignore                                                          0.1s
 => => transferring context: 2B                                                                    0.0s
 => [default internal] load metadata for docker.io/library/ubuntu:latest                           0.5s
 => [default 1/4] FROM docker.io/library/ubuntu:latest@sha256:e3f92abc0967a6c19d0dfa2d55838833e94  0.0s
 => CACHED [default 2/4] RUN apt-get update &&     apt-get install -y bash python3 python3-pip ps  0.0s
 => CACHED [default 3/4] RUN ln -s /usr/bin/python3 /usr/bin/python                                0.0s
 => CACHED [default 4/4] RUN pip3 install pyshark scapy pwntools dpkt pycryptodome cryptography o  0.0s
 => [default] exporting to image                                                                   0.0s
 => => exporting layers                                                                            0.0s
 => => writing image sha256:71d9aff9ab808ca81f66b51873752131da8e9c2378617ec4a045703814ce6e86       0.0s
 => => naming to docker.io/library/inspect-intercode_ctf-wzucacvyd7anv6yhe7adp6-default            0.0s

╭─ intercode_ctf (1 sample) ───────────────────────────────────────────────────────────────────────────╮
│ shuffle: True, max_messages: 30                                         mistral/mistral-large-latest │
│                                                                                      dataset: ic_ctf │
│ total time:                        0:00:20                                          scorer: includes │
│ mistral/mistral-large-latest       2,490 tokens [2,365 + 125]                                        │
│                                                                                                      │
│ accuracy: 0  bootstrap_std: 0                                                                        │
│                                                                                                      │
│ Log: ./logs/2024-06-17T14-35-05+00-00_intercode-ctf_RzdyPcZBvwPDiSxUQ4BH7m.json                      │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────╯
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/pathlib.py", line 1116, in mkdir
    os.mkdir(self, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/run/user/1000/inspect_ai/view'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/pathlib.py", line 1116, in mkdir
    os.mkdir(self, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/run/user/1000/inspect_ai'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/pathlib.py", line 1116, in mkdir
    os.mkdir(self, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/run/user/1000'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/vscode/.local/bin/inspect", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/workspaces/inspect_ai/src/inspect_ai/_cli/main.py", line 47, in main
    inspect(auto_envvar_prefix="INSPECT")
  File "/home/vscode/.local/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/inspect_ai/src/inspect_ai/_cli/common.py", line 30, in wrapper
    return cast(click.Context, func(*args, **kwargs))
                               ^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/inspect_ai/src/inspect_ai/_cli/common.py", line 55, in wrapper
    return cast(click.Context, func(*args, **kwargs))
                               ^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/inspect_ai/src/inspect_ai/_cli/eval.py", line 238, in eval_command
    eval(
  File "/workspaces/inspect_ai/src/inspect_ai/_eval/eval.py", line 102, in eval
    return asyncio.run(
           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/workspaces/inspect_ai/src/inspect_ai/_eval/eval.py", line 301, in eval_async
    view_notify_eval(logger.location)
  File "/workspaces/inspect_ai/src/inspect_ai/_view/view.py", line 221, in view_notify_eval
    file = view_last_eval_file()
           ^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/inspect_ai/src/inspect_ai/_view/view.py", line 252, in view_last_eval_file
    return view_runtime_dir() / "last-eval-result"
           ^^^^^^^^^^^^^^^^^^
  File "/workspaces/inspect_ai/src/inspect_ai/_view/view.py", line 248, in view_runtime_dir
    return inspect_runtime_dir("view")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/inspect_ai/src/inspect_ai/_util/appdirs.py", line 12, in inspect_runtime_dir
    runtime_dir.mkdir(parents=True, exist_ok=True)
  File "/usr/local/lib/python3.11/pathlib.py", line 1120, in mkdir
    self.parent.mkdir(parents=True, exist_ok=True)
  File "/usr/local/lib/python3.11/pathlib.py", line 1120, in mkdir
    self.parent.mkdir(parents=True, exist_ok=True)
  File "/usr/local/lib/python3.11/pathlib.py", line 1120, in mkdir
    self.parent.mkdir(parents=True, exist_ok=True)
  File "/usr/local/lib/python3.11/pathlib.py", line 1116, in mkdir
    os.mkdir(self, mode)
PermissionError: [Errno 13] Permission denied: '/run/user'

[jjallaire]

I think the issue is that the devcontainer doesn't have the XDG environment variables set (not that it should!) and the Python platformsdirs library doesn't provide defaults (I'd assumed it would). I think we just need to provide these at startup if they don't exist. We'd use the default values as outlined here https://platformdirs.readthedocs.io/en/latest/api.html#unix-linux

[jjallaire]

Can you check a couple of things:

1) What does platformdirs.user_runtime_path("inspect_ai") return in the dev container?

2) Is that directory writeable?

3) What is the $HOME in the devcontainer?

Note that a lot of software relies on these vars, so I might also argue that devcontainers that don't export them aren't well set up for development. That said, we should provide defaults that work if they are absent.

[alex-treebeard]

Thanks for the help

1

Python 3.11.4 (main, Jun  7 2023, 18:32:58) [GCC 10.2.1 20210110]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.25.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import platformdirs

In [2]: platformdirs.user_runtime_path("inspect_ai") Out[2]: PosixPath('/run/user/1000/inspect_ai')

> 2.
Not to the vscode user I'm running as

> 3.
`/home/vscode`

I've worked around this by running (and added it to my Dockerfile

```sh
sudo mkdir -p /run/user/1000
sudo chown vscode:vscode /run/user/1000
sudo chmod 700 /run/user/1000

[craigwalton-dsit]

Just to add an alternative workaround to this: export XDG_RUNTIME_DIR=/tmp.