search

erikbra / grate

grate - the SQL scripts migration runner
MIT License
165 stars 36 forks source link

Error with Grate 1.7.0 - Read-only file system #509

Closed mbp closed 1 month ago

mbp commented 2 months ago

Since the version 1.7.0 of Grate, we are getting errors when running it in our Docker environment.

Log file:

Looking in /tmp/tmphW0TFr.tmp for scripts to run. An error occurred: Read-only file system : '/app/output'

We use AWS Fargate to run grate with readonlyrootFileSystem enabled for security reasons. It seems new grate version is writing to the root file system. Can this change be reverted?

This issue can also be reproduced if you run grate in a Docker with --read-only set, e.g.

docker run -t my-grate-migration-container --read-only
erikbra commented 2 months ago

I understand why this is happening. It's the internal grate migration that extracts embedded SQL from the DLL and writes them to a temporary folder before running an extra, internal grate migration on its own internal tables, to make sure they are up-to-date, before proceeding with the actual migration. This is due to #482 . I did think of it, but I thought it was an exotic use case to not have a writable temporary folder. Of course I was wrong 😛

I'll try to look into virtualising the fil system access, so that we are not dependent on a writable file system for this. In the mean time, I recommend rolling back to grate 1.6.2.

A related question, how do you handle grate's changeDropFolder/outputPath? Do you get a lot of warnings that we're unable to copy the scripts to the changeDropFolder?

erikbra commented 2 months ago

A follow-up question, @mbp, could you try to add a tmpfs volume on /tmp, and see if that solves you issue? https://docs.docker.com/storage/tmpfs/ or is that not possible in your situation?

mbp commented 2 months ago

@erikbra right now we have workaround with pinning the version to 1.6.2. So there is no urgent issue for us 👍

We do have writable tmp folder. It's just the root file system that is not writable (I suppose it's to prevent someone from modfying any application code)

We actually invoke Docker like so:

docker run -t my-grate-migration-container --read-only --tmpfs "/tmp"

And grate is invoked like so:

grate --dbt postgresql -ct 300 --silent -f /tmp/UpgradeFiles -o /tmp/grate -cs "XXX"

As you can see, we set output path to tmp folder. I am not sure what is meant by "changeDropFolder", but we don't get any warnings when running grate.