search

nipy / heudiconv

Flexible DICOM conversion into structured directory layouts
https://heudiconv.readthedocs.io
Other
234 stars 125 forks source link

Can't rebuild joss paper #725

Closed TheChymera closed 8 months ago

TheChymera commented 8 months ago

just in case you forgot, the current state of the paper is here (last commit from me trying podman since docker was failing):

[deco]~/heudiconv ❱ git rev-parse HEAD
a6e87d432d26554b17abe924a053205ddf7f8169
[deco]~/heudiconv ❱ git branch
  master
* paper-joss

Sadly, both still fail, any ideas why that might be? @asmacdo @yarikoptic 

[deco]~/heudiconv ❱ make paper OCI_BINARY=podman
podman run --rm --volume /home/chymera/heudiconv:/data:Z --user 1000:1000 --env JOURNAL=joss openjournals/inara
Error: short-name "openjournals/inara" did not resolve to an alias and no unqualified-search registries are defined in "/etc/containers/registries.conf"
make: *** [Makefile:5: paper] Error 125
[deco]~/heudiconv ❱ make paper
docker run --rm --volume /home/chymera/heudiconv:/data:Z --user 1000:1000 --env JOURNAL=joss openjournals/inara
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error setting cgroup config for procHooks process: bpf_prog_query(BPF_CGROUP_DEVICE) failed: invalid argument: unknown.
make: *** [Makefile:5: paper] Error 125
asmacdo commented 8 months ago

@TheChymera not sure what's going on with docker, but my guess is a local environment issue? can you do docker run hello-world?

For podman you need a fully qualified image name. If docker (but not podman) receives a "short name", (ie openjournals/inara) docker will assume the registry is docker.io.

podman run --rm --volume /home/chymera/heudiconv:/data:Z --user 1000:1000 --env JOURNAL=joss docker.io/openjournals/inara

asmacdo commented 8 months ago

For the docker issue, if hello-world isn't working, what's your kernel? https://github.com/docker/cli/issues/3481

TheChymera commented 8 months ago

Docker hello world also fails :(

[deco]~ ❱ docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
c1ec31eb5944: Pull complete
Digest: sha256:ac69084025c660510933cca701f615283cdbb3aa0963188770b54c31c8962493
Status: Downloaded newer image for hello-world:latest
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error setting cgroup config for procHooks process: bpf_prog_query(BPF_CGROUP_DEVICE) failed: invalid argument: unknown.
ERRO[0002] error waiting for container:
TheChymera commented 8 months ago

The link you posted doesn't work, will try again in a few minutes, maybe something with the website 🤔 Do you mean that docker requires specific kernel config? Yes, I know, but I already mutilated my kernel to support docker and it worked for the longest time. I have a bunch of kernel options missing, but I don't think those are hard requirements. Do you have everything under “generally necessary”?

[deco]~ ❱ /usr/share/docker/contrib/check-config.sh
info: reading kernel config from /proc/config.gz ...

Generally Necessary:
- cgroup hierarchy: cgroupv2
  Controllers:
  - cpu: available
  - cpuset: available
  - io: available
  - memory: available
  - pids: available
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: enabled
- CONFIG_CGROUP_FREEZER: enabled
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_CPUSETS: enabled
- CONFIG_MEMCG: enabled
- CONFIG_KEYS: enabled
- CONFIG_VETH: enabled
- CONFIG_BRIDGE: enabled
- CONFIG_BRIDGE_NETFILTER: missing
- CONFIG_IP_NF_FILTER: enabled
- CONFIG_IP_NF_MANGLE: enabled
- CONFIG_IP_NF_TARGET_MASQUERADE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled
- CONFIG_NETFILTER_XT_MATCH_IPVS: missing
- CONFIG_NETFILTER_XT_MARK: enabled (as module)
- CONFIG_IP_NF_NAT: enabled (as module)
- CONFIG_NF_NAT: enabled
- CONFIG_POSIX_MQUEUE: enabled
- CONFIG_CGROUP_BPF: missing

Optional Features:
- CONFIG_USER_NS: enabled
- CONFIG_SECCOMP: enabled
- CONFIG_SECCOMP_FILTER: enabled
- CONFIG_CGROUP_PIDS: enabled
- CONFIG_MEMCG_SWAP: missing
    (cgroup swap accounting is currently enabled)
- CONFIG_BLK_CGROUP: enabled
- CONFIG_BLK_DEV_THROTTLING: missing
- CONFIG_CGROUP_PERF: missing
- CONFIG_CGROUP_HUGETLB: missing
- CONFIG_NET_CLS_CGROUP: missing
- CONFIG_CGROUP_NET_PRIO: missing
- CONFIG_CFS_BANDWIDTH: missing
- CONFIG_FAIR_GROUP_SCHED: enabled
- CONFIG_IP_NF_TARGET_REDIRECT: missing
- CONFIG_IP_VS: enabled
- CONFIG_IP_VS_NFCT: missing
- CONFIG_IP_VS_PROTO_TCP: missing
- CONFIG_IP_VS_PROTO_UDP: missing
- CONFIG_IP_VS_RR: missing
- CONFIG_SECURITY_SELINUX: enabled
- CONFIG_SECURITY_APPARMOR: missing
- CONFIG_EXT4_FS: enabled
- CONFIG_EXT4_FS_POSIX_ACL: enabled
- CONFIG_EXT4_FS_SECURITY: enabled
- Network Drivers:
  - "overlay":
    - CONFIG_VXLAN: missing
    - CONFIG_BRIDGE_VLAN_FILTERING: missing
      Optional (for encrypted networks):
      - CONFIG_CRYPTO: enabled
      - CONFIG_CRYPTO_AEAD: enabled
      - CONFIG_CRYPTO_GCM: enabled
      - CONFIG_CRYPTO_SEQIV: enabled
      - CONFIG_CRYPTO_GHASH: enabled
      - CONFIG_XFRM: enabled
      - CONFIG_XFRM_USER: enabled
      - CONFIG_XFRM_ALGO: enabled
      - CONFIG_INET_ESP: missing
      - CONFIG_NETFILTER_XT_MATCH_BPF: missing
  - "ipvlan":
    - CONFIG_IPVLAN: missing
  - "macvlan":
    - CONFIG_MACVLAN: missing
    - CONFIG_DUMMY: missing
  - "ftp,tftp client in container":
    - CONFIG_NF_NAT_FTP: enabled
    - CONFIG_NF_CONNTRACK_FTP: enabled
    - CONFIG_NF_NAT_TFTP: missing
    - CONFIG_NF_CONNTRACK_TFTP: missing
- Storage Drivers:
  - "btrfs":
    - CONFIG_BTRFS_FS: enabled
    - CONFIG_BTRFS_FS_POSIX_ACL: missing
  - "devicemapper":
    - CONFIG_BLK_DEV_DM: enabled
    - CONFIG_DM_THIN_PROVISIONING: missing
  - "overlay":
    - CONFIG_OVERLAY_FS: missing
  - "zfs":
    - /dev/zfs: missing
    - zfs command: missing
    - zpool command: missing

Limits:
- /proc/sys/kernel/keys/root_maxkeys: 1000000
TheChymera commented 8 months ago

Ok, maybe docker now needs kernel things it didn't need before...

As for podman, ok, that worked, but there's another issue. Any idea what that might be about?

[deco]~/src/heudiconv ❱ make paper OCI_BINARY=podman
podman run --rm --volume /home/chymera/src/heudiconv:/data:Z --user 1000:1000 --env JOURNAL=joss docker.io/openjournals/inara
Trying to pull docker.io/openjournals/inara:latest...
Getting image source signatures
Copying blob ef5531b6e74e done   |
Copying blob 4e821479e4b5 done   |
Copying blob c6b7e74dbfd1 done   |
Copying blob 13d63ff5f1df done   |
Copying blob 00f1e069c9e8 done   |
Copying blob 73732c6ec5a1 done   |
Copying blob 984e9b553bec done   |
Copying blob 36f506d57a55 done   |
Copying blob b4a7ca1217f5 done   |
Copying blob 057ddd0af801 done   |
Copying blob 0e8fc7fd216f done   |
Copying blob 7a9e5ae8d256 done   |
Copying blob 1d75b44bd9c0 done   |
Copying blob 64c6d2e78959 done   |
Copying blob be77d8be8bdf done   |
Copying blob 4e38139f4e33 done   |
Copying blob 6ce2d5edaa3e done   |
Copying blob 34b9ad025475 done   |
Copying blob d2f200484d6f done   |
Copying blob 8d1611934a3f done   |
Copying blob 2fa8e81e1d76 done   |
Copying config 3a121e8852 done   |
Writing manifest to image destination
[INFO] Running filter /usr/local/share/openjournals/data/filters/parse-latex.lua
[INFO] Completed filter /usr/local/share/openjournals/data/filters/parse-latex.lua in 24 ms
[INFO] Running filter /usr/local/share/openjournals/data/filters/inline-cited-references.lua
[INFO] Loaded paper.bib from paper.bib
[INFO] Completed filter /usr/local/share/openjournals/data/filters/inline-cited-references.lua in 56 ms
[INFO] Running filter citeproc
[INFO] Loaded apa.csl from /usr/local/share/openjournals/apa.csl
[INFO] Completed filter citeproc in 187 ms
[INFO] Running filter /usr/local/share/openjournals/data/filters/normalize-metadata.lua
[INFO] Completed filter /usr/local/share/openjournals/data/filters/normalize-metadata.lua in 30 ms
[INFO] Running filter /usr/local/share/openjournals/data/filters/time.lua
[INFO] Completed filter /usr/local/share/openjournals/data/filters/time.lua in 16 ms
[INFO] Running filter /usr/local/share/openjournals/data/filters/normalize-author-names.lua
[INFO] Completed filter /usr/local/share/openjournals/data/filters/normalize-author-names.lua in 25 ms
[INFO] Running filter /usr/local/share/openjournals/data/filters/substitute-in-format.lua
[INFO] Completed filter /usr/local/share/openjournals/data/filters/substitute-in-format.lua in 12 ms
[INFO] Running filter /usr/local/share/openjournals/data/filters/resolve-references.lua
[INFO] Completed filter /usr/local/share/openjournals/data/filters/resolve-references.lua in 40 ms
[INFO] Running filter /usr/local/share/openjournals/data/filters/remove-references-heading.lua
[INFO] Completed filter /usr/local/share/openjournals/data/filters/remove-references-heading.lua in 13 ms
[INFO] Running filter /usr/local/share/openjournals/data/filters/orcid-uri.lua
[INFO] Completed filter /usr/local/share/openjournals/data/filters/orcid-uri.lua in 48 ms
[INFO] Loaded figs/environment.pdf from figs/environment.pdf
[INFO] Loaded figs/workflow.pdf from figs/workflow.pdf
[INFO] Loaded figs/usage.pdf from figs/usage.pdf
pandoc: media: createDirectory: permission denied (Permission denied)
make: *** [Makefile:5: paper] Error 1
[deco]~/src/heudiconv ❱ ls paper.pdf
ls: cannot access 'paper.pdf': No such file or directory
yarikoptic commented 8 months ago

FWIW with OCI_BINARY=podman I am also having problems. Works fine with docker.

TheChymera commented 8 months ago

Tested it out locally as well, the linked PR fixes the podman build.