Change folder of "execution"

[thaarhoff]

Hi,

in our project, we want to use rio cogeo create daily within a k8s cluster. This is currently set up, by having a cronjob, deploying a custom docker container(unix-based), which is linked to a PVC. Let's say it's mounted to /opt/images From this location the job runs rio cogeo create --cog-profile deflate --blocksize 256 --overview-level 8 --overview-resampling bilinear source_file target_file What we noticed is, during loading the input file, the /tmp directory rapidly increases. Up to a point where the container gets evicted when processing large files.

I couldn't find a suitable configuration option to change the folder to point to the PVC mount at /opt/images. Is this possible somehow?

(Sure - I could change the mount, but the job also is used for other processes which are configured to the current mount and it would be prefered to tell rio cogeo where to run)

Greetings

[vincentsarago]

@thaarhoff is your source_file a local file?

maybe you can try to change CPL_TMPDIR environment variable to point to a specific directory

is. target_file in /tmp/....

by default we will create a temporary file in the same directory as the target destination https://github.com/cogeotiff/rio-cogeo/blob/main/rio_cogeo/cogeo.py#L28-L40

[thaarhoff]

Thanks for the quick response. rio is executed in /opt/images. The source file is also in that folder. The target file would be in a (already existing) subfolder.

/opt/images < execution /opt/images/source_file /opt/images/result/target_file

I'll set the environment variable and come back.

[thaarhoff]

Before rio execution

/# du --max-depth=1
68028460        ./opt
8       ./tmp

echo $CPL_TMPDIR > /opt/geoimages/result

Starting rio Reading input: /opt/geoimages/input.tif and tmp starts rising again.

/# du --max-depth=1
68028460        ./opt
918124  ./tmp

[vincentsarago]

And what's in tmp ?

[thaarhoff]

Before, tmp is empty. There is a tmp-tif file being generated

And that is not the filename of the target_file

[vincentsarago]

Interesting So the file is created in tmp

Can you make sure that --allow-intermediate-compression is used

[thaarhoff]

I added the option. I mean - it raises slower. Still the file is created in tmp. 3.4GB at the moment.

And evicted at 12GB, since /vda1 is full. While the PVC with 400GB is untouched :D

[vincentsarago]

🤦 I see! I guess in https://github.com/cogeotiff/rio-cogeo/blob/main/rio_cogeo/cogeo.py#L33, is_file() will only return True when we pass a valid (existing) file.

I'll fix this and release a new version 🙏

[thaarhoff]

Thank you for investigating! I'll look forward to it.

[vincentsarago]

the origin bug should be fixed in 3.5.1 (which should be on pypi in couple minutes). I'll open another PR later to add a new option for user to pass a temporary directory

[thaarhoff]

Works. Thank you for the quick fix <3