input-output-hk / cicero

event-driven automation for Nomad
https://handbook.cicero.ci.iog.io
41 stars 11 forks source link

Nomad Jobs failing with "OCI permission denied" #69

Closed jhbertra closed 1 year ago

jhbertra commented 1 year ago

So far, all runs of this action https://cicero.ci.iog.io/action/b4767439-ff93-4926-941b-89c0065b5251 which have successfully been invoked are failing on Nomad with an "OCI permission denied" error. I would copy the logs, but #58. Here is an example run https://cicero.ci.iog.io/run/f65a60a3-9fed-46e5-9d6b-c9afd5976966

jhbertra commented 1 year ago

I see this is also happening on https://cicero.ci.iog.io/action/51321349-4140-4cdf-80b4-0702822896c5

Anviking commented 1 year ago

We're also seeing this in some cardano-wallet builds, e.g. https://cicero.ci.iog.io/run/af6fdd4a-01b8-491d-90f6-d8adcf3ab3dd

dermetfan commented 1 year ago

This is an issue with podman which seems to corrupt its zfs state from time to time... We're trying to get rid of it

dermetfan commented 1 year ago

Closing as exec driver is now working and we are migrating actions from podman to exec one by one.