Ok, this one i huge. But there are just many changes need to make this work.
I think the most important are:
Separation of docker and docker compose files for the API and the processor to be able to manage both in one repo while running them independently on different servers
COG and thumbnail generation can now run on a separate processing server. Needed files are pulled from the storage server and results are pushed back. All files on the processing servers are stored in a temporary folder, to avoid any conflicts by accidentally running the process on the storage server.
new process for cog_processing which pulls tifs and pushes cogs back to storage server
new procee_thumbnail with the same logic.
updated processing states, added processing and errored for both thumbnail and cog. I have kept processing and errored to be backwards compatible.
new generic route for all processes. It can be started for cog and thumbnail individual or for both. I wanted to be able to share the pulled TIF between processes, so I needed a way to manage a process which runs all processes together while still having the flexibility to run them individually.
configuring dev or production tables via environmental variables. So, together with the updated docker and docker compose files there are some changes to how to deploy the API, @mmaelicke lets have a small chat.
There are logs of code changes caused by the formatter :)
I think this is a nightmare to review, therefore just skim though, and if some things are unclear or entirely wrong or dangerous, comment it. Otherwise, I would try to deploy it and iterate on possible errors.
Ok, this one i huge. But there are just many changes need to make this work.
I think the most important are:
cog_processingwhich pulls tifs and pushes cogs back to storage serverprocee_thumbnailwith the same logic.processinganderroredto be backwards compatible.There are logs of code changes caused by the formatter :) I think this is a nightmare to review, therefore just skim though, and if some things are unclear or entirely wrong or dangerous, comment it. Otherwise, I would try to deploy it and iterate on possible errors.