Read-only file system error on Lambda

sklink commented 1 year ago

I've successfully deployed the function to Lambda but am getting this error when running a test in the AWS Lambda dashboard:

AWS_LAMBDA_RUNTIME_API = 127.0.0.1:9001, running Julia
OpenBLAS WARNING - could not determine the L2 cache size on this system, assuming 256k
ERROR: InitError: AWS_LAMBDA_RUNTIME_API = 127.0.0.1:9001, running Julia
OpenBLAS WARNING - could not determine the L2 cache size on this system, assuming 256k
ERROR: InitError: SystemError: opening file "/var/runtime/julia_depot/logs/scratch_usage.toml": Read-only file system
Stacktrace:
[1] systemerror(p::String, errno::Int32; extrainfo::Nothing)
@ Base ./error.jl:176
[2] #systemerror#80
@ ./error.jl:175 [inlined]
[3] systemerror
@ ./error.jl:175 [inlined]
[4] open(fname::String; lock::Bool, read::Nothing, write::Nothing, create::Nothing, truncate::Nothing, append::Bool)
@ Base ./iostream.jl:293
[5] open(f::Scratch.var"#4#7"{String}, args::String; kwargs::Base.Pairs{Symbol, Bool, Tuple{Symbol}, NamedTuple{(:append,), Tuple{Bool}}})
@ Base ./io.jl:382
[6] track_scratch_access(pkg_uuid::Base.UUID, scratch_path::String)
@ Scratch /var/runtime/julia_depot/packages/Scratch/u24Rl/src/Scratch.jl:174
[7] get_scratch!(parent_pkg::Base.UUID, key::String, calling_pkg::Base.UUID)
@ Scratch /var/runtime/julia_depot/packages/Scratch/u24Rl/src/Scratch.jl:236
[8] macro expansion
@ /var/runtime/julia_depot/packages/Scratch/u24Rl/src/Scratch.jl:302 [inlined]
[9] _scratch_dir
@ /var/runtime/julia_depot/packages/TimeZones/2oXvO/src/TimeZones.jl:34 [inlined]
[10] __init__()
@ TimeZones.TZData /var/runtime/julia_depot/packages/TimeZones/2oXvO/src/tzdata/TZData.jl:20
[11] _include_from_serialized(pkg::Base.PkgId, path::String, depmods::Vector{Any})
@ Base ./loading.jl:831
[12] _tryrequire_from_serialized(modkey::Base.PkgId, path::String, sourcepath::String, depmods::Vector{Any})
@ Base ./loading.jl:938
[13] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String, build_id::UInt64)
@ Base ./loading.jl:1028
[14] _require(pkg::Base.PkgId)
@ Base ./loading.jl:1315
[15] _require_prelocked(uuidkey::Base.PkgId)
@ Base ./loading.jl:1200
[16] macro expansion
@ ./loading.jl:1180 [inlined]
[17] macro expansion
@ ./lock.jl:223 [inlined]
[18] require(into::Module, mod::Symbol)
@ Base ./loading.jl:1144
during initialization of module TZData
START RequestId: e94387be-016d-4451-a3c5-c1a16eb318b9 Version: $LATEST
RequestId: e94387be-016d-4451-a3c5-c1a16eb318b9 Error: Runtime exited with error: exit status 1
Runtime.ExitError
END RequestId: e94387be-016d-4451-a3c5-c1a16eb318b9
REPORT RequestId: e94387be-016d-4451-a3c5-c1a16eb318b9  Duration: 10262.32 ms   Billed Duration: 10263 ms   Memory Size: 2000 MB    Max Memory Used: 496 MB

So far I've tried to add the --privileged param to docker run. I also tried adding extra permissions to the IAM Role that was created by Jot.jl.

This build is coming from a GitHub runner.

Any suggestions?

sklink commented 1 year ago

Attempted to add RUN chmod 775 /var/runtime/julia_depot -R to the Jot Dockerfile with no luck.

harris-chris commented 1 year ago

That's the first thing I would have tried as well.

For my debugging purposes, which Julia version are you using? I updated Jot to use 1.82 by default when I made that fix last week, so if you're using the defaults then it's probably that.

I don't think I'm going to get any time on this in the next 24 hours but I'll look at it this week.

On Wed, 2 Nov 2022, 04:48 Matt Doak, @.***> wrote:

Attempted to add RUN chmod 775 /var/runtime/julia_depot -R to the Jot Dockerfile with no luck.

— Reply to this email directly, view it on GitHub https://github.com/harris-chris/Jot.jl/issues/35#issuecomment-1299039225, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALDMVSHUAHUT6VUS7VF4BPLWGFX2PANCNFSM6AAAAAARUKSE5Y . You are receiving this because you are subscribed to this thread.Message ID: @.***>

harris-chris commented 1 year ago

Also, is there anything special about the responder function you're using? When we spoke last week, were you able to get the simple example from the documentation to run on AWS Lambda ok?

On Wed, 2 Nov 2022, 04:48 Matt Doak, @.***> wrote:

Attempted to add RUN chmod 775 /var/runtime/julia_depot -R to the Jot Dockerfile with no luck.

— Reply to this email directly, view it on GitHub https://github.com/harris-chris/Jot.jl/issues/35#issuecomment-1299039225, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALDMVSHUAHUT6VUS7VF4BPLWGFX2PANCNFSM6AAAAAARUKSE5Y . You are receiving this because you are subscribed to this thread.Message ID: @.***>

harris-chris commented 1 year ago

Seems like it might be related to this: https://stackoverflow.com/questions/66698034/aws-lambda-read-only-file-system-error-using-docker-image-to-store-ml-model

On Wed, 2 Nov 2022, 04:48 Matt Doak, @.***> wrote:

Attempted to add RUN chmod 775 /var/runtime/julia_depot -R to the Jot Dockerfile with no luck.

— Reply to this email directly, view it on GitHub https://github.com/harris-chris/Jot.jl/issues/35#issuecomment-1299039225, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALDMVSHUAHUT6VUS7VF4BPLWGFX2PANCNFSM6AAAAAARUKSE5Y . You are receiving this because you are subscribed to this thread.Message ID: @.***>

sklink commented 1 year ago

Yes, I landed on that as well. It's the Scratch.jl package that's attempting to write so I'm looking for a way to either point it elsewhere or remove the related dependency (NPFinancial)

Thanks for the quick response. Will update if I find something.

sklink commented 1 year ago

Ended up adding this to your Dockerfile which worked:

RUN ln -s /var/runtime/julia_depot/logs/ /tmp/julia_depot/logs/

Some sort of before deploy hook into the Jot generated Dockerfile would cover this and some of the scenarios with binary issues. I did attempt to do that instead of writing it directly in the BuildDockerfile.jl but had no luck getting it to pick up the string I would send.

We're okay running with our forked version for now. I could come back and attempt the Docker hook again as a PR once I'm done with this deliverable. Feel free to close this if you'd like.

harris-chris commented 1 year ago

Yes, that's a clever solution. I'm surprised this hasn't been a problem before, I guess it's rare for packages to want to write to the julia_depot. I think a hook to allow arbitrary text to be added to the Dockerfile would be a good feature, but it would also be nice to have Jot ensure that the julia_depot is writeable anyway.

On Wed, 2 Nov 2022, 06:30 Matt Doak, @.***> wrote:

Ended up adding this to your Dockerfile which worked:

RUN ln -s /var/runtime/julia_depot/logs/ /tmp/julia_depot/logs/

Some sort of before deploy hook into the Jot generated Dockerfile would cover this and some of the scenarios with binary issues. I did attempt to do that instead of writing it directly in the BuildDockerfile.jl but had no luck getting it to pick up the string I would send.

We're okay running with our forked version for now. I could come back and attempt the Docker hook again as a PR once I'm done with this deliverable. Feel free to close this if you'd like.

— Reply to this email directly, view it on GitHub https://github.com/harris-chris/Jot.jl/issues/35#issuecomment-1299217608, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALDMVSHIWEEVLBVJLKLPKYDWGGDXZANCNFSM6AAAAAARUKSE5Y . You are receiving this because you commented.Message ID: <harris-chris/Jot .@.***>

harris-chris commented 1 year ago

I've been trying to implement this now, but I can't seem to square the circle. The problem as I see it is that:

The /tmp folder is the only writable folder at run-time, therefore where Scratch.jl needs to write to.
Scratch.jl seems to always write the Julia depot path, it can't be paramaterized to work any other way
The AWS documentation seems vague on when the /tmp folder gets emptied, but from testing it seems that it's empty when the function runs (even if it's had stuff added to it in the Dockerfile)
So we can't put the julia depot folder in the /tmp path, as it won't be there at runtime

Could I check this fits with your understanding? If you managed to make it work, where did you add the

RUN ln -s /var/runtime/julia_depot/logs/ /tmp/julia_depot/logs/

line above?

harris-chris commented 1 year ago

Actually never mind, I think I have this working. You can enter multiple paths for the depot path.

harris-chris commented 1 year ago

This should now be fixed in the most recent release - the DEPOT_PATH is writable.

harris-chris / Jot.jl

Read-only file system error on Lambda #35