Max File Size Setting

[sparton175]

I managed to get the standalone TouchTerrain server running on my Ubuntu 22.04 server and I'm able to use it just like the public site, including the size limits of the files. I was hoping with this standalone servers I could generate files above the limit of the public site to create very large 3d printed maps. Where can I find the setting to change the size limit? And what would be a reasonable limit to set it to so I don't crash my system or fill my SSD in one go, this is a fairly beefy system, 12 xeon gold cores and 32gb of ram with a 100gb NVMe SSD.

[ChHarding]

Hi!

You seemed to have solved this on your own but maybe this is still helpful:

• in our sever we’ve set a quota (which is enforced before the actual processing starts) that is really only based on what we know our (small-ish) server can run w/o running out of memory. The setting for this is in touch terrain/server/config.py but as you’re running standalone this won’t affect you.
• The only potentially interesting setting is max_cells_for_memory_only, which you would have to set as an argument for get_zipped_tiles (something like args["max_cells_for_memory_only”] = 10000000 which would attempt to run the processing fully in memory and then write it to a file rather than write piecemeal to a file during processing. Note that I’ve not pushed this to the limit and I can’t remember if there also need to be additional temp “space” in memory during processing. So I don’t really know how large a raster you could in-memory process with your 32 Gbs.
• Also, multi-processing (CPU_cores_to_use = 0) may or may not work. It used to work but it then broke for our server at one point, so I disabled it on it.
• BTW the 10 Megapixels are BTW a hard limit that Google Earth Engine imposes as download limit for it’s tiff files, so you can only go beyond that if you download a raster to your Google drive manually (see en of the ReadMe.md on how that works) and use that as importeedDEM
• Can I ask how you’re using the standalone version? As Jupiter notebook? If so you might be interest in using colab with this notebook:

https://github.com/ChHarding/TouchTerrain_for_CAGEO/blob/master/TouchTerrain_jupyter_starters_colab.ipynb

Which I’m currently testing as a much simpler alternative to setting up standalone. Just click the badge (or got to https://colab.research.google.com/github/ChHarding/TouchTerrain_for_CAGEO/blob/master/TouchTerrain_jupyter_starters_colab.ipynb ) to install everything on colab. If you happen to run it please let me know what you think as I’m thinking of advertising this as an easy way to run standalone.

Finally, can I ask what you’re creating the terrain models for?

Cheers

Chris

On Oct 9, 2022, at 01:17, sparton175 @.**@.>> wrote:

I managed to get the standalone TouchTerrain server running on my Ubuntu 22.04 server and I'm able to use it just like the public site, including the size limits of the files. I was hoping with this standalone servers I could generate files above the limit of the public site to create very large 3d printed maps. Where can I find the setting to change the size limit? And what would be a reasonable limit to set it to so I don't crash my system or fill my SSD in one go, this is a fairly beefy system, 12 xeon gold cores and 32gb of ram with a 100gb NVMe SSD.

— Reply to this email directly, view it on GitHubhttps://github.com/ChHarding/TouchTerrain_for_CAGEO/issues/59, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AEYDF5NNVB7OU6I5M5EDG4LWCJPQXANCNFSM6AAAAAARAR2BB4. You are receiving this because you are subscribed to this thread.Message ID: @.***>

Chris Harding Associate Professor Department of Geological & Atmospheric Sciences Touchterrain.geol.iastate.eduhttp://Touchterrain.geol.iastate.edu

[nstew21]

I managed to get the standalone TouchTerrain server running on my Ubuntu 22.04 server and I'm able to use it just like the public site, including the size limits of the files. I was hoping with this standalone servers I could generate files above the limit of the public site to create very large 3d printed maps. Where can I find the setting to change the size limit? And what would be a reasonable limit to set it to so I don't crash my system or fill my SSD in one go, this is a fairly beefy system, 12 xeon gold cores and 32gb of ram with a 100gb NVMe SSD.

Hello, I'm currently trying to solve this same problem, and would love it if you could fill us in on how you managed to do it. <3

[ChHarding]

you could massively increase the MAX_CELLS_PERMITTED and MAX_CELLS values in touchterrain/server/config.py

limits for ISU server

for STL/OBJ don't even start with a DEM bigger than that number. GeoTiff export is this * 100!

MAX_CELLS_PERMITED = 1000 1000 4 # private

MAX_CELLS_PERMITED = 1000 1000 0.7

if DEM has > this number of cells, use tempfile instead of memory

MAX_CELLS = MAX_CELLS_PERMITED / 4

On Apr 6, 2024, at 08:29, nstew21 @.**@.>> wrote:

I managed to get the standalone TouchTerrain server running on my Ubuntu 22.04 server and I'm able to use it just like the public site, including the size limits of the files. I was hoping with this standalone servers I could generate files above the limit of the public site to create very large 3d printed maps. Where can I find the setting to change the size limit? And what would be a reasonable limit to set it to so I don't crash my system or fill my SSD in one go, this is a fairly beefy system, 12 xeon gold cores and 32gb of ram with a 100gb NVMe SSD.

Hello, I'm currently trying to solve this same problem, and would love it if you could fill us in on how you managed to do it. <3

— Reply to this email directly, view it on GitHubhttps://github.com/ChHarding/TouchTerrain_for_CAGEO/issues/59#issuecomment-2041082481, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AEYDF5INZKDJI3DQ6QNT3HDY372DZAVCNFSM6AAAAAARAR2BB6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANBRGA4DENBYGE. You are receiving this because you commented.Message ID: @.***>

Chris Harding Associate Professor Department of Geological & Atmospheric Sciences Touchterrain.geol.iastate.eduhttp://Touchterrain.geol.iastate.edu

[nstew21]

Hi Chris,

I seem to be struggling massively with this. I am trying to run your jupyter notebook for starters, and am just trying to generate anything at this point prior to adjusting touchterrain/server/config.py file.

I do however assume editing it as such would immediately increase the size, any pointers or immediate issues you see?

limits for ISU server for STL/OBJ don't even start with a DEM bigger than that number. GeoTiff export is this * 100!

MAX_CELLS_PERMITED = 1000 1000 4 # private

MAX_CELLS_PERMITED = 10000 10000 0.7

if DEM has > this number of cells, use tempfile instead of memory MAX_CELLS = MAX_CELLS_PERMITED / 4

Then of course my struggles with even generating anything without modifying a thing... I am stuck on step 3 apparently, Importing python packages, but potentially because of a failed ee.Authenticate(). I'm not too sure.

I run it once, it seems to work. I generate a token, paste it in and it says Successfully saved authorization token.

Next step however, and it feeds out an error, which once I work my way down the page seems to result in it not working. Any tips?

Importing python packages Run the next cell, it'll import more python packages, including Touchterrain and give you some initial print settings The earth engine package you imported and authorized earlier, will be initialized here. You should see EE init() worked with .config/earthengine/credentials

import packages import os.path from glob import glob import k3d from random import randint from shutil import rmtree import zipfile from geojson import Polygon import geemap from touchterrain.common import TouchTerrainEarthEngine as TouchTerrain from touchterrain.common.TouchTerrainGPX import * args = TouchTerrain.initial_args # default args WARNING:root:EE init() error (with .config/earthengine/credentials) ('invalid_scope: Bad Request', {'error': 'invalid_scope', 'error_description': 'Bad Request'}) (This is OK if you don't use earthengine anyway!)

Clearly I don't understand this well enough to realize what is wrong and would appreciate any help. I tried this a while back and I think it worked originally, and when I went through the process this morning it actually seemed to have already been run and successful. I clear it out, delete the container in docker and try to reset but it consistently fails now.

Thanks, Nathan

[nstew21]

However when I run the TouchTerrain_standalone_jupyter_notebook and it seems to be happier and actually generated a file, so perhaps crisis averted. Quite the thing you've built here, much appreciated.

[ChHarding]

My apologies for the issues with gaming the size check. I’ve create a version of the TouchTerrain_app.py file that only issues a warning but does allow any size/number of tiles to be processed. Make a copy go you existing TouchTerrain_app.py file (TouchTerrain_app.py.orig), download this file

https://github.com/ChHarding/TouchTerrain_for_CAGEO/blob/no_size_limit/touchterrain/server/TouchTerrain_app.py

And copy it to TouchTerrain_for_CAGEO/touchterrain/server

However when I run the TouchTerrain_standalone_jupyter_notebook and it seems to be happier and actually generated a file, so perhaps crisis averted. Quite the thing you've built here, much appreciated.

If you a OK with using Python, I would recommend using the standalone notebook. It has a lot more functionality and even has a Google Maps Style area selector.

— Reply to this email directly, view it on GitHubhttps://github.com/ChHarding/TouchTerrain_for_CAGEO/issues/59#issuecomment-2053761515, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AEYDF5NBKQNZTSTJ5JK44ZLY5GM6FAVCNFSM6AAAAAARAR2BB6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANJTG43DCNJRGU. You are receiving this because you commented.Message ID: @.***>

Chris Harding Associate Professor Department of Geological & Atmospheric Sciences Touchterrain.geol.iastate.eduhttp://Touchterrain.geol.iastate.edu