Local install (non Docker) causing Storage to fill up

[anciltech]

I ran the "pip install ." with the setup.py and all other files in the working directory of Python, and my Mac started asking me for permission to access Photos, Contacts etc, witch I found odd, and noticed that the write speed to my local storage was very high and filling up my SSD. I terminated the code and only had a few gigs left on the drive. After a restart the storage returned back to where it was before I attempted the install. What the heck is all that about? Im new to python, but I dont think I did anything that drastically wrong. I have been working with the docker version beforehand, but wanted to see if there was any better performance in the undockerized version.

[ChHarding]

Hi,

pip install . only downloads and installs the required python packages including all their prerequisites. If you’ve started with a vanilla python installation that could be a lot but you’d have seen pip commandline messages as they get installed. No sure if that would hammer your SSD or how much disk space that needs. Maybe a couple of gigs? If you started with something like Anaconda (which I do), you already have most of the large packages installed. My current Anaconda folder is about 5Gb although I have installed a lot of other packages in the last year that are not needed for TouchTerrain so I imagine a fresh Anaconda install + whatever TouchTerrain needs would be less(?)

Also, I do all my development on Win 10 b/c have actually never managed to run TouchTerrain standalone on my Mac. I never could get OsGeo/GDAL installed w/o compiling stuff for it (didn’t want to go home-brew or install a C-compiler …).

But, just for my interest, I completely uninstalled and re-installed Docker and pulled the image: docker pull chharding/touchterrain_jupyter After the pull, I had about 5 Gb less disk space (2Gb download, expands to 5Gb). And that's just the docker image, the install_touchterrain.sh script will clone entire GitHub repo and then run pip install . on that. But I’m unclear if that results in even less disk space on your SSD or if that ends up “inside” the 5Gb container. Sorry, I’m really not very familiar with Docker!

Also, although I’ve never timed it properly, I don’t think I’ve noticed a big performance loss when using the docker version. Granted, that only applies to my Win 10 PC where I routinely run the non-docker version for debugging, etc. Can I ask why you’re exploring the non-docker version? Do you simply want to save the overhead disk space?

I would recommend sticking with the docker version b/c I’m really not sure if it’s even possible to run the non-docker version on Mac b/c of the OsGeo/GDAL install issues I run into. That was ab out 1.5 years ago on an older MacOS version, so maybe that’s not an issue anymore. But I’d hate for you to waste a lot of time (like I did) on this only to end up having to fall back to docker anyway!

If you want to try it I would get rid of the docker image and also uninstall Docker. Then start your (local) pip install . and watch the disk space in About this Mac - Storage (or maybe there’s a unix util for that?).

Sorry, I know that’s a lot of conjecture and speculation but that’s all I can think of! However, I can assure you that this is not some sort of crypto mining BS that gets maliciously installed :) (Although that’s ofc what any bad actor would say ….)

Final question, can I ask what you’re trying to do with TouchTerrain?

Cheers

Chris

On Sep 18, 2021, at 10:03 AM, anciltech @.***> wrote:

I ran the "pip install ." with the setup.py and all other files in the working directory of Python, and my Mac started asking me for permission to access Photos, Contacts etc, witch I found odd, and noticed that the write speed to my local storage was very high and filling up my SSD. I terminated the code and only had a few gigs left on the drive. After a restart the storage returned back to where it was before I attempted the install. What the heck is all that about? Im new to python, but I dont think I did anything that drastically wrong. I have been working with the docker version beforehand, but wanted to see if there was any better performance in the undockerized version.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/ChHarding/TouchTerrain_for_CAGEO/issues/53, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEYDF5NWXJ4UDGVUJI32ONLUCSS43ANCNFSM5EJJFKZQ. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

Chris Harding Associate Professor Department of Geological & Atmospheric Sciences Touchterrain.geol.iastate.edu

[anciltech]

Thanks for the detailed response Chris, it filled up about 150Gb of space so a little bit more than I think a normal install would take ;)

I am using a resin printer to create some cool wall art and gifts, and the printer really capture every triangle face if it isn't high enough quality, ergo large file sizes and running into some of the size limits, I was looking to run the local code for increased performance and hopefully success rate. Sometimes the docker container version will slow if the files are large, until it eventually doesn't show any progress or CPU/GPU resources used; any idea why that is?

Seems like you guys are running quite the rig behind the web version, that'll spit out a 150Gb file size in about 30 seconds where it takes quite a while on my Laptop. Maybe ill stick with that! Thanks for the tool, this makes things possible that I'd never be able to figure out on my own.

[ChHarding]

Ah, then I have a hunch! TouchTerrain works by first processing the DEM (this is when you see 10% …. 90%) during which it accumulates data into the tmp folder. That’s done b/c for large models this would not fit into memory, so the temp folder is just used as a sort of slow memory dump. (BTW there’s a config setting that defines when a raster is too large to be processed fully in memory, that way small rasters can be processed pretty quickly.) TT then goes over that temp file (you’ll see a message about assembling triangles) and converts it into a STL (inside a dedicated folder) and the zips that folder.

But, in your case something in your data or setup must lead to these bonkers large temp files. I suspect you’re doing large and/or very high resolution rasters (maybe even more than one in parallel?). How many pixels is your geotiff that you import? Or are you getting online DEM data? What is your print resolution set to? Are you using the original raster resolution (i.e. -1) or are you setting it so a very small value (< 0.1) MY experience with SLA is limited but we should talk about what the practical limit for you case would be (i.e. below that it doesn’t give you a better print, just more processing). For typical FDM that’s around 0.4 mm (maybe down to 0.2 mmm) b/c that’s the smallest blob that a 0.4 mm nozzle can create. For example, assume you have the absolute highest peak of a steep mountain. It won’t matter that that peek is covered by 10 x 10 cells in your DEM raster, the sliver will have to amalgamate those 100 elevation values into a single value b/c that’s the size of the smallest blob that can come out of the nozzle. So it would be smarter to down-sample that DEM before processing, so that instead of 10 x 10 pixels, we’d get just 1 x 1 pixel for that peak. I suspect that that’s at the core of why you’re getting these huge files. Note that this x/y resolution has nothing to d with your fabulously small SLA layer heights! (Well, there some linkage but it’s complicated and the effect is very small :)

So what I would like you to do is to do a test run with a lower resolution or a smaller raster and watch both the output for the 2 processing stage and your tmp folder. Let’s establish if my thinking is correct, then we can talk about finding an effective print resolution for your printer, OK?

Cheers

Chris

On Sep 19, 2021, at 12:29 PM, anciltech @.***> wrote:

Thanks for the detailed response Chris, it filled up about 150Gb of space so a little bit more than I think a normal install would take ;)

I am using a resin printer to create some cool wall art and gifts, and the printer really capture every triangle face if it isn't high enough quality, ergo large file sizes and running into some of the size limits, I was looking to run the local code for increased performance and hopefully success rate. Sometimes the docker container version will slow if the files are large, until it eventually doesn't show any progress or CPU/GPU resources used; any idea why that is?

Seems like you guys are running quite the rig behind the web version, that'll spit out a 150Gb file size in about 30 seconds where it takes quite a while on my Laptop. Maybe ill stick with that! Thanks for the tool, this makes things possible that I'd never be able to figure out on my own.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/ChHarding/TouchTerrain_for_CAGEO/issues/53#issuecomment-922508756, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEYDF5LK7KFI4LBQWCTNGZDUCYMXPANCNFSM5EJJFKZQ. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

Chris Harding Associate Professor Department of Geological & Atmospheric Sciences Touchterrain.geol.iastate.edu

[ChHarding]

So did this resolve your issue?

On Sep 19, 2021, at 12:29 PM, anciltech @.***> wrote:

Thanks for the detailed response Chris, it filled up about 150Gb of space so a little bit more than I think a normal install would take ;)

I am using a resin printer to create some cool wall art and gifts, and the printer really capture every triangle face if it isn't high enough quality, ergo large file sizes and running into some of the size limits, I was looking to run the local code for increased performance and hopefully success rate. Sometimes the docker container version will slow if the files are large, until it eventually doesn't show any progress or CPU/GPU resources used; any idea why that is?

Seems like you guys are running quite the rig behind the web version, that'll spit out a 150Gb file size in about 30 seconds where it takes quite a while on my Laptop. Maybe ill stick with that! Thanks for the tool, this makes things possible that I'd never be able to figure out on my own.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/ChHarding/TouchTerrain_for_CAGEO/issues/53#issuecomment-922508756, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEYDF5LK7KFI4LBQWCTNGZDUCYMXPANCNFSM5EJJFKZQ. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

Chris Harding Associate Professor Department of Geological & Atmospheric Sciences Touchterrain.geol.iastate.edu

[anciltech]

Hi Chris, sorry for the long reply time, in the middle of moving at the moment.

Setting the "max_cells_for_memory_only" to 10^20 definitely did help, and I'm using a 4x2 grid to utilize my 8 core processor, then combining and solidifying the models together afterwards with Meshmixer.

I am still able to get it to get TT stuck though, when increasing the resolution further. I am using online DEM data and usually start at a .4 resolution to see how it looks, and then go as far down as I can go with the resolution before running into the TT stall issue, which typically occurs around a 0.1 and a total size of 240x180mm. I just tried setting the resolution to -1 but an error returns with "EEException: Pixel grid dimensions (14210x10725) must be less than or equal to 10000."

As far as the validity of going to such low resolutions, SLA horizontal resolution is 0.05mm and the Z can be set to 0.01mm, which in my experience means that you can really see each face of each triangle from up close, and it makes the print look "out of focus" from further back. Also, steep faces, even with high resolution, can end up coming out jagged (see attached photo of Half Dome face at a resolution of 0.2, log attached for reference). What I've found can help is the Solidify function in Meshmixer, which can help smooth out some of those artifacts while removing leftover edges and structures from the previous 4x2 grid.

What do you think? Am I asking too much?

logfile.txt

[ChHarding]

Hi Chris, sorry for the long reply time, in the middle of moving at the moment.

Setting the "max_cells_for_memory_only" to 10^20 definitely did help, and I'm using a 4x2 grid to utilize my 8 core processor, then combining and solidifying the models together afterwards with Meshmixer.

I am still able to get it to get TT stuck though, when increasing the resolution further. I am using online DEM data and usually start at a .4 resolution to see how it looks, and then go as far down as I can go with the resolution before running into the TT stall issue, which typically occurs around a 0.1 and a total size of 240x180mm. I just tried setting the resolution to -1 but an error returns with "EEException: Pixel grid dimensions (14210x10725) must be less than or equal to 10000."

Well, there you’re running into a Google server restriction. It simply won’t serve geotiffs that large. There are 3 ways around that:

1) Knowing the corner coordinates you can “download” very large geotiff from Google Earth Engine via a javascript tool that transfers the geotiff into your Google drive from which you can download it then manually and use it in TouchTerrain standalone. This is described at the very end of this ReadMe: https://github.com/ChHarding/TouchTerrain_for_CAGEO#readme https://github.com/ChHarding/TouchTerrain_for_CAGEO#readme.

2) Split your area into multiple tiles, say 2 x 2 = 4 tiles. Each tile should now be below the 10,000 pixel limit, if not use more tiles. Use the only setting in Manual settings to only process one tile at a time: E.g. “only”:[1,1] will only process tile x=1, y=1, “only”:[2,1] for x=2, y=1, etc. Unzip each and put the 4 stills into meshmixer (with append). They will “fit” together on the virtual build plate (i.e. touch each other to form a rectangle) and you can simple print those 4 parts together. The only downside is that you’ll still have the internal walls, so that’s a try bit of waste. Maybe Meshmixer could remove this but I never bothered to find out.

But, there seems to be a hardware limit on our server that users can’t get over, even if they manage to get below 10k pixels. I know that b/c I now have a 400 x 400 mm printer (CR-6 max) and with 0.4 mm nozzle I can never get 400 x 400 mm through, it always times out. Something like 380 x 380 works but not 400 x 400. According to my server guy its just a RAM limitation and we’re stuck with whatever we have for now. So, on to ...

3) Again using tiles, but only download the geotiffs, not the STLs and then run standalone on them. Either merge the geotiff into a single geotiff (using QGIS) or just create 4 STLs and append them into 1 model in Meshmixer.

But I would suggest that you first increase your 0.2 mm to whatever makes you lower the pixel count below 10000 and see if that goes through. I think that would still look very nice on SLA with 0.1 mm layer height.

Cheers

Chris

[anciltech]

Hi Chris

I've done some testing over the past day on the different options above to see which would be the best...

1. Downloading TIF from EE - This also stops in the middle of processing when the file is at a similar resolution to what I've tried before. An interesting tangent though, pulling the TIF at the original resolution of 30m (var scale: 30 during export of DSM from JAXA/ALOS/AW3D30/V2_2) results in a far rougher model than setting the resolution to 5m, as shown in the screenshots. How is this possible if the original data is only accurate to 30m?

   

2. Manually looping with "Only" - This works and seems like the best option at the moment, but is slow, since it only uses one core. Also, Meshmixer is able to remove these inner walls with the Make Solid tool, which works quite well. I am hollowing out my prints to save on resin so I have been doing this after combining.

3. Using TT to obtain tiled TIFs - I only ran into the 10000 limit when setting the resolution to -1. I dont think Ill need full resolution at any point, but I'll look into this in the future.

About the failure during processing, do you think it could be an issue with the multiprocessing? I was looking through your code a bit and was thinking maybe this synchronization implementation might be the trick ... https://docs.python.org/3/library/multiprocessing.html#synchronization-between-processes

Thanks for all the help! Max

PS Ive updated Docker and TT isn't able to start anymore. Ive deleted and reinstalled TT, and tried to Trust the notebook, but it always leads to: [W 14:29:05.983 NotebookApp](B Notebook TouchTerrain_jupyter_for_starters.ipynb is not trusted Operation not permitted (src/thread.cpp:309) Operation not permitted (src/thread.cpp:309) qemu: uncaught target signal 6 (Aborted) - core dumped ./run_touchterrain.sh: line 7: 53 Aborted jupyter notebook --ip 0.0.0.0 --port 8888 --no-browser --allow-root root@a33de40e914d:/TouchTerrain# Operation not permitted (src/thread.cpp:309) Operation not permitted (src/thread.cpp:309) qemu: uncaught target signal 6 (Aborted) - core dumped

[ChHarding]

Max,

Let’s talk about the notebook issue first. I’ve now also upgraded to Docker 4.01 and have reinstalled TouchTerrain. This works fine for me:

It does say notebook not trusted in the CLI and in the notebook but that still lets me use the nb as usual, even the geemap cell works. I guess I don’t understand what Javascript disabled means in this context. If I hit the Trust button it restarts the nb in trusted mode. I guess I have no idea right now why you’re getting this: Operation not permitted (src/thread.cpp:309)

Is it possible that we have different versions of Jupyter despite using the same install script?

BTW I used the 127.0.0.1:8888 url in my browser.

Is it possible that something in your browser changed recently? What are you using?

Cheers

Chris

PS Ive updated Docker and TT isn't able to start anymore. Ive deleted and reinstalled TT, and tried to Trust the notebook, but it always leads to: [W 14:29:05.983 NotebookApp]�(B Notebook TouchTerrain_jupyter_for_starters.ipynb is not trusted Operation not permitted (src/thread.cpp:309) Operation not permitted (src/thread.cpp:309) qemu: uncaught target signal 6 (Aborted) - core dumped ./run_touchterrain.sh: line 7: 53 Aborted jupyter notebook --ip 0.0.0.0 --port 8888 --no-browser --allow-root @.***:/TouchTerrain# Operation not permitted (src/thread.cpp:309) Operation not permitted (src/thread.cpp:309) qemu: uncaught target signal 6 (Aborted) - core dumped

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/ChHarding/TouchTerrain_for_CAGEO/issues/53#issuecomment-936479582, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEYDF5ISGCEHZJGQ4VJJW6DUFRSCVANCNFSM5EJJFKZQ. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[anciltech]

I did some digging after you said your nb was untrusted as well, and it appears to be a variable name change in a library (See log below). The following https://github.com/ipython-contrib/jupyter_contrib_nbextensions/issues/1529 suggests to downgrade nbconvert to less than v6, although that didn't work, so ill have to go and change the variable names in a few places. I'll give this a shot later and see if it works. Just not sure why this all changed, do you?

Ive been using the same, 127.0.0.1. Always just copy and paste the last link. I usually use safari. I tried Chrome now too with the same result. Ive also tried running pip install --upgrade jupyterlab. It seemed to be directly tied to the Docker update, but I have been installing other things which I dont understand, like conda and some other tools, but nothing in the docker environment.

The following occurs after opening the TouchTerrain_standalone_jupyter_notebook.ipynb file...

[I 14:28:15.440 NotebookApp](B 302 GET /?token=eff3b58a32e239fe28397a413c0fca19871795fea0e047ea (172.17.0.1) 8.480000ms [W 14:29:02.845 NotebookApp](B Config option template_path not recognized by ExporterCollapsibleHeadings. Did you mean one of: extra_template_paths, template_name, template_paths? [W 14:29:02.888 NotebookApp](B Config option template_path not recognized by ExporterCollapsibleHeadings. Did you mean one of: extra_template_paths, template_name, template_paths? [W 14:29:03.006 NotebookApp](B Config option template_path not recognized by TocExporter. Did you mean one of: extra_template_paths, template_name, template_paths? [W 14:29:03.035 NotebookApp](B Config option template_path not recognized by TocExporter. Did you mean one of: extra_template_paths, template_name, template_paths? [W 14:29:03.086 NotebookApp](B Config option template_path not recognized by LenvsHTMLExporter. Did you mean one of: extra_template_paths, template_name, template_paths? [W 14:29:03.177 NotebookApp](B Config option template_path not recognized by LenvsHTMLExporter. Did you mean one of: extra_template_paths, template_name, template_paths? [W 14:29:03.241 NotebookApp](B Config option template_path not recognized by LenvsTocHTMLExporter. Did you mean one of: extra_template_paths, template_name, template_paths? [W 14:29:03.288 NotebookApp](B Config option template_path not recognized by LenvsTocHTMLExporter. Did you mean one of: extra_template_paths, template_name, template_paths? [W 14:29:03.426 NotebookApp](B Config option template_path not recognized by LenvsLatexExporter. Did you mean one of: extra_template_paths, template_name, template_paths? [W 14:29:03.453 NotebookApp](B Config option template_path not recognized by LenvsLatexExporter. Did you mean one of: extra_template_paths, template_name, template_paths? [W 14:29:04.015 NotebookApp](B Config option template_path not recognized by LenvsSlidesExporter. Did you mean one of: extra_template_paths, template_name, template_paths? [W 14:29:04.045 NotebookApp](B Config option template_path not recognized by LenvsSlidesExporter. Did you mean one of: extra_template_paths, template_name, template_paths? [W 14:29:04.369 NotebookApp](B Config option template_path not recognized by ExporterCollapsibleHeadings. Did you mean one of: extra_template_paths, template_name, template_paths? [W 14:29:04.407 NotebookApp](B Config option template_path not recognized by ExporterCollapsibleHeadings. Did you mean one of: extra_template_paths, template_name, template_paths? [W 14:29:04.520 NotebookApp](B Config option template_path not recognized by TocExporter. Did you mean one of: extra_template_paths, template_name, template_paths? [W 14:29:04.548 NotebookApp](B Config option template_path not recognized by TocExporter. Did you mean one of: extra_template_paths, template_name, template_paths? [W 14:29:04.589 NotebookApp](B Config option template_path not recognized by LenvsHTMLExporter. Did you mean one of: extra_template_paths, template_name, template_paths? [W 14:29:04.636 NotebookApp](B Config option template_path not recognized by LenvsHTMLExporter. Did you mean one of: extra_template_paths, template_name, template_paths? [W 14:29:04.696 NotebookApp](B Config option template_path not recognized by LenvsTocHTMLExporter. Did you mean one of: extra_template_paths, template_name, template_paths? [W 14:29:04.743 NotebookApp](B Config option template_path not recognized by LenvsTocHTMLExporter. Did you mean one of: extra_template_paths, template_name, template_paths? [W 14:29:04.871 NotebookApp](B Config option template_path not recognized by LenvsLatexExporter. Did you mean one of: extra_template_paths, template_name, template_paths? [W 14:29:04.898 NotebookApp](B Config option template_path not recognized by LenvsLatexExporter. Did you mean one of: extra_template_paths, template_name, template_paths? [W 14:29:05.466 NotebookApp](B Config option template_path not recognized by LenvsSlidesExporter. Did you mean one of: extra_template_paths, template_name, template_paths? [W 14:29:05.495 NotebookApp](B Config option template_path not recognized by LenvsSlidesExporter. Did you mean one of: extra_template_paths, template_name, template_paths? [I 14:29:05.975 NotebookApp](B Writing notebook-signing key to /root/.local/share/jupyter/notebook_secret [W 14:29:05.983 NotebookApp](B Notebook TouchTerrain_jupyter_for_starters.ipynb is not trusted Operation not permitted (src/thread.cpp:309) Operation not permitted (src/thread.cpp:309) qemu: uncaught target signal 6 (Aborted) - core dumped ./run_touchterrain.sh: line 7: 53 Aborted jupyter notebook --ip 0.0.0.0 --port 8888 --no-browser --allow-root root@a33de40e914d:/TouchTerrain# Operation not permitted (src/thread.cpp:309) Operation not permitted (src/thread.cpp:309) qemu: uncaught target signal 6 (Aborted) - core dumped exit root@a33de40e914d:/TouchTerrain# exit

[ChHarding]

I did some digging after you said your nb was untrusted as well, and it appears to be a variable name change in a library (See log below). The following ipython-contrib/jupyter_contrib_nbextensions#1529 https://github.com/ipython-contrib/jupyter_contrib_nbextensions/issues/1529 suggests to downgrade nbconvert to less than v6, although that didn't work, so ill have to go and change the variable names in a few places. I'll give this a shot later and see if it works. Just not sure why this all changed, do you?

Just read about the nbconvert version thing. You should try it but I’d use pip and not condo (not sure it’s actually installed on my docker image)

pip uninstall nbconvert …..that worked

Pip install nbconvert==5.6.1. also technically worked but is broke some stuff:

I uninstall all the offending packages and reinstalled them as per their requirement, all except voila. After that my notebook still ran, even the gee map cell.

So, maybe try this?

Cheers

Chris

[ChHarding]

Max,

Just wanted to try and answer your other points (besides the jupyter weirdness …).Please let me know if and how you manage to clear that up.

On Oct 6, 2021, at 10:18 AM, anciltech @.***> wrote:

Hi Chris

I've done some testing over the past day on the different options above to see which would be the best...

Downloading TIF from EE - This also stops in the middle of processing when the file is at a similar resolution to what I've tried before.

How big is that tif? And at what resolution does it stop working?

An interesting tangent though, pulling the TIF at the original resolution of 30m (var scale: 30 during export of DSM from JAXA/ALOS/AW3D30/V2_2) results in a far rougher model than setting the resolution to 5m, as shown in the screenshots. How is this possible if the original data is only accurate to 30m? So you resampled the 30 m DEM to 5 m, which is done via a linear interpolation (in this case), which tends to smooth or blur:

https://developers.google.com/earth-engine/guides/resample https://developers.google.com/earth-engine/guides/resample

Note that this example shows bicubic interpolation which is even smoother than linear. If the resampling were to use nearest neighbor, your 5 m DEM would look very close to the 30 m DEM b/c multiple 5 m sample would have the exact same value as the closest 30 m sample value.

https://user-images.githubusercontent.com/79978026/136229973-2c196400-de82-4fe3-ac6a-fcf8c1845f1e.png https://user-images.githubusercontent.com/79978026/136230039-455de3e8-3ad2-4fd8-8500-8e36e865268c.png Manually looping with "Only" - This works and seems like the best option at the moment, but is slow, since it only uses one core. Also, Meshmixer is able to remove these inner walls with the Make Solid tool, which works quite well. I am hollowing out my prints to save on resin so I have been doing this after combining.

Well, so that’s not ideal but seems to give you a solution for now.

Using TT to obtain tiled TIFs - I only ran into the 10000 limit when setting the resolution to -1. I dont think Ill need full resolution at any point, but I'll look into this in the future.

About the failure during processing, do you think it could be an issue with the multiprocessing? I was looking through your code a bit and was thinking maybe this synchronization implementation might be the trick ... https://docs.python.org/3/library/multiprocessing.html#synchronization-between-processes https://docs.python.org/3/library/multiprocessing.html#synchronization-between-processesUnless you create tiled models the code will run in Single Core mode. If you look at log file it will say using single-code only somewhere. Thanks for all the help! Max

PS Ive updated Docker and TT isn't able to start anymore. Ive deleted and reinstalled TT, and tried to Trust the notebook, but it always leads to: [W 14:29:05.983 NotebookApp]�(B Notebook TouchTerrain_jupyter_for_starters.ipynb is not trusted Operation not permitted (src/thread.cpp:309) Operation not permitted (src/thread.cpp:309) qemu: uncaught target signal 6 (Aborted) - core dumped ./run_touchterrain.sh: line 7: 53 Aborted jupyter notebook --ip 0.0.0.0 --port 8888 --no-browser --allow-root @.***:/TouchTerrain# Operation not permitted (src/thread.cpp:309) Operation not permitted (src/thread.cpp:309) qemu: uncaught target signal 6 (Aborted) - core dumped

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/ChHarding/TouchTerrain_for_CAGEO/issues/53#issuecomment-936479582, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEYDF5ISGCEHZJGQ4VJJW6DUFRSCVANCNFSM5EJJFKZQ. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[anciltech]

Hi Chris, I've been trying to fix this issue but haven't had much luck. I'm pretty new to python, docker and Jupyter, but the closest I've gotten is copied below. The 302 GET is me opening Jupyter link in a browser, and once I click on on of the Notebooks, I get these Operation not permitted errors. Have you run into this before? It might be a M1 Mac issue, as I can't find much about this around google. Just strange that it was working fine before.

[I 17:19:44.124 NotebookApp] 302 GET /?token=e86d032641ca6ba1b98647aa5accc5068d39b4d1cf321213 (172.17.0.1) 6.570000ms Operation not permitted (src/thread.cpp:309) qemu: uncaught target signal 6 (Aborted) - core dumped Aborted

Operation not permitted (src/thread.cpp:309)

Operation not permitted (src/thread.cpp:309) qemu: uncaught target signal 6 (Aborted) - core dumped

[ChHarding]

Max,

I’m sorry but I can’t help you with this. My Mac mini is a pre-M1 2019 model and just doesn’t have that issue. I agree that it's very strange that is just happened out of nowhere. I’m assuming that jupyter didn’t get a new version that causes this b/c then more users would have this pretty massive issue. So your M1 hypothesis is the best I can think of. Maybe MacOS suddenly decided changed how threading and permissions work? I’m wondering if whatever causes this is running as a Python wrapper around C++ and maybe that would have to be complied specifically for M1 hardware? But, I’m really not into C++ so this is pure speculation ...

However, I do have some potentially good news: For the new Touchterrain v3.5 I’ve rewritten how STL/OBJ files are created and have substantially lowered how much memory is used (I basically dump stuff to disk as it’s being created rather than putting everything in memory first and then dumping it …). I’m hoping that avoids the timeout that sometimes happened to users of the web app where it would run out of memory and kill the entire process. Not sure if that was ever an issue for you? Maybe this would enable you to use the web app? This would also apply to the standalone version, if you can get it to run. Remind me what was the issue for not running your standalone directly on your Mac rather via docker?

If you need to use docker but can’t use jupyter, how about using the JSON file based standalone version? You would have hand edit the example_config.json file for your areas coordinates, print settings etc. that’s in the stuff folder copy it to project root and then run TouchTerrain_standalone.py with it:

python TouchTerrain_standalone.py myarea.json

Problem is that even if that works I don’t know how to “export” the resulting zip file from docker to your Mac. As this is typically done vi Jupyter’s web interface I never investigated that I’m this is the first thing I’ve used docker for ever. But maybe that’s something worth investigating?

Cheers

Chris

On Oct 24, 2021, at 12:51, anciltech @.***> wrote:

Hi Chris, I've been trying to fix this issue but haven't had much luck. I'm pretty new to python, docker and Jupyter, but the closest I've gotten is copied below. The 302 GET is me opening Jupyter link in a browser, and once I click on on of the Notebooks, I get these Operation not permitted errors. Have you run into this before? It might be a M1 Mac issue, as I can't find much about this around google. Just strange that it was working fine before.

[I 17:19:44.124 NotebookApp] 302 GET /?token=e86d032641ca6ba1b98647aa5accc5068d39b4d1cf321213 (172.17.0.1) 6.570000ms Operation not permitted (src/thread.cpp:309) qemu: uncaught target signal 6 (Aborted) - core dumped Aborted

Operation not permitted (src/thread.cpp:309)

Operation not permitted (src/thread.cpp:309) qemu: uncaught target signal 6 (Aborted) - core dumped

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

[anciltech]

Hi Chris, Finally some good news:

Docker Version: I did the obvious and just downgraded Docker from 4.1.1 to 4.0.0, now everything is working (not sure why I didn't just try this first). I noted this issue under the Docker issue https://github.com/docker/for-mac/issues/6016#issuecomment-950837498

Local Jupyter Version: Great to hear! Let me know if you need any help testing for future releases. I downloaded v3.5 and gave a local install another try. I had more luck this time, but had to mess around with a import gdal issue (not totally sure how I fixed it, but FYI I received this error after initial setup >>> import gdal return _load(spec) ImportError: dlopen(/Users/Genesis/opt/miniconda3/lib/python3.9/site-packages/osgeo/_gdal.cpython-39-darwin.so, 0x0002): Library not loaded: @rpath/libpoppler.91.dylib)

The processing on the local version is much faster for me than the Docker version. At least 2x, which I am loving, and TT has been able to create 700+Mb files with ease, although I am only able to do a single tile. I get an error with multiple tiles. The following error comes after all tiles have been created:

Error sending result: '[({'DEMname': 'JAXA-ALOS-AW3D30-V2_2', 'crs': 'EPSG:32632', 'UTMzone': 'UTM 32N', 'scale': 125122.9924536569, 'pixel_mm': 0.08982035928143713, 'max_elev': 21.29301683616701, 'min_elev': 4.749294859834901, 'z_scale': 1, 'base_thickness_mm': 2, 'bottom_relief_mm': 1.0, 'folder_name': 'V2_2_9.17_46.82', 'tile_centered': False, 'tile_no_x': 2, 'tile_no_y': 1, 'tile_width': 60.0, 'tile_height': 90.9880239520958, 'full_raster_width': 2674, 'full_raster_height': 2028, 'fileformat': 'STLb', 'temp_file': None, 'no_bottom': False, 'bottom_image': None, 'ntilesy': 2, 'only': None, 'no_normals': True, 'geo_transform': (498173.0736371183, 11.23859213655601, 0.0, 5197073.400299733, 0.0, -11.23859213655601), 'use_geo_coords': None, 'smooth_borders': True, 'file_size': 129.38793563842773}, <memory at 0x7f8a7860aa00>)]'. Reason: 'TypeError("cannot pickle 'memoryview' object")' processing tile: 4 2 top min/max: 2.0 19.96632222357226 ... multi-core processing done, logging resumed

4 x 2 tiles, tile size 60.00 x 90.99 mm

---

UnboundLocalError Traceback (most recent call last) /var/folders/5k/99c1_1ks5mg432mzv8m22ds40000gn/T/ipykernel_30812/758848299.py in ----> 1 totalsize, full_zip_file_name = TouchTerrain.get_zipped_tiles(**args) # args are in a dict 2 print("\nDONE!\n\nCreated zip file", full_zip_file_name, "%.2f" % totalsize, "Mb")

/Users/Genesis/Downloads/TouchTerrain_for_CAGEO-master-2/touchterrain/common/TouchTerrainEarthEngine.py in get_zipped_tiles(DEM_name, trlat, trlon, bllat, bllon, polygon, polyURL, poly_file, importedDEM, printres, ntilesx, ntilesy, tilewidth, basethick, zscale, fileformat, tile_centered, CPU_cores_to_use, max_cells_for_memory_only, temp_folder, zip_file_name, no_bottom, bottom_image, ignore_leq, lower_leq, unprojected, only, original_query_string, no_normals, projection, use_geo_coords, importedGPX, gpxPathHeight, gpxPixelsBetweenPoints, gpxPathThickness, map_img_filename, smooth_borders, offset_masks_lower, fill_holes, **otherargs) 1402 # concat all processed tiles into a zip file 1403 #print "start of putting tiles into zip file") -> 1404 for p in processed_list: 1405 tile_info = p[0] # per-tile info 1406 tn = DEM_title+"tile%d_%d" % (tile_info["tile_no_x"],tile_info["tile_no_y"]) + "." + fileformat[:3] # name of file inside zip

UnboundLocalError: local variable 'processed_list' referenced before assignment

Local Python Version: I receive the same error above under the same condition, the only difference being I ran the commands directly through python.

[ChHarding]

Hi Chris, Finally some good news:

Docker Version: I did the obvious and just downgraded Docker from 4.1.1 to 4.0.0, now everything is working (not sure why I didn't just try this first). I noted this issue under the Docker issue docker/for-mac#6016 (comment https://github.com/docker/for-mac/issues/6016#issuecomment-950837498Well done! I didn’t think if that either :) Hopefully they’ll fix that eventually. At least now there’s a solution if other M1 users report this to me. Local Version: Great to hear! Let me know if you need any help testing for future releases. I downloaded v3.5 and gave a local install another try. I had more luck this time, but had to mess around with a import gdal issue (not totally sure how I fixed it, but FYI I received this error after initial setup >>> import gdal return _load(spec) ImportError: dlopen(/Users/Genesis/opt/miniconda3/lib/python3.9/site-packages/osgeo/_gdal.cpython-39-darwin.so, 0x0002): Library not loaded: @rpath/libpoppler.91.dylib)

GDAL/OsGeo seems to be a terrible mess to install. I tried it on my previous Mac and gave up on getting a prebuild of it (which exist for Windows). Would have had to get a C compiler and compile to from source, which was too much for me. This is what prompted me to look into Docker The processing on the local version is much faster for me than the Docker version. At least 2x, which I am loving, and TT has been able to create 700+Mb files with ease, although I am only able to do a single tile. I get an error with multiple tiles. The following error comes after all tiles have been created:

Error sending result: '[({'DEMname': 'JAXA-ALOS-AW3D30-V2_2', 'crs': 'EPSG:32632', 'UTMzone': 'UTM 32N', 'scale': 125122.9924536569, 'pixel_mm': 0.08982035928143713, 'max_elev': 21.29301683616701, 'min_elev': 4.749294859834901, 'z_scale': 1, 'base_thickness_mm': 2, 'bottom_relief_mm': 1.0, 'folder_name': 'V2_2_9.17_46.82', 'tile_centered': False, 'tile_no_x': 2, 'tile_no_y': 1, 'tile_width': 60.0, 'tile_height': 90.9880239520958, 'full_raster_width': 2674, 'full_raster_height': 2028, 'fileformat': 'STLb', 'temp_file': None, 'no_bottom': False, 'bottom_image': None, 'ntilesy': 2, 'only': None, 'no_normals': True, 'geo_transform': (498173.0736371183, 11.23859213655601, 0.0, 5197073.400299733, 0.0, -11.23859213655601), 'use_geo_coords': None, 'smooth_borders': True, 'file_size': 129.38793563842773}, <memory at 0x7f8a7860aa00>)]'. Reason: 'TypeError("cannot pickle 'memoryview' object")' processing tile: 4 2 top min/max: 2.0 19.96632222357226 ... multi-core processing done, logging resumed

4 x 2 tiles, tile size 60.00 x 90.99 mm

UnboundLocalError Traceback (most recent call last) /var/folders/5k/99c1_1ks5mg432mzv8m22ds40000gn/T/ipykernel_30812/758848299.py in ----> 1 totalsize, full_zip_file_name = TouchTerrain.get_zipped_tiles(**args) # args are in a dict 2 print("\nDONE!\n\nCreated zip file", full_zip_file_name, "%.2f" % totalsize, "Mb")

/Users/Genesis/Downloads/TouchTerrain_for_CAGEO-master-2/touchterrain/common/TouchTerrainEarthEngine.py in get_zipped_tiles(DEM_name, trlat, trlon, bllat, bllon, polygon, polyURL, poly_file, importedDEM, printres, ntilesx, ntilesy, tilewidth, basethick, zscale, fileformat, tile_centered, CPU_cores_to_use, max_cells_for_memory_only, temp_folder, zip_file_name, no_bottom, bottom_image, ignore_leq, lower_leq, unprojected, only, original_query_string, no_normals, projection, use_geo_coords, importedGPX, gpxPathHeight, gpxPixelsBetweenPoints, gpxPathThickness, map_img_filename, smooth_borders, offset_masks_lower, fill_holes, **otherargs) 1402 # concat all processed tiles into a zip file 1403 #print "start of putting tiles into zip file") -> 1404 for p in processed_list: 1405 tile_info = p[0] # per-tile info 1406 tn = DEMtitle+"tile%d%d" % (tile_info["tile_no_x"],tile_info["tile_no_y"]) + "." + fileformat[:3] # name of file inside zip

UnboundLocalError: local variable 'processed_list' referenced before assignmen

That’s a lot to unpack and I’m not sure I know python well enough for this. I’m guessing it has to do with multiprocessing. I’m going to see if I broke something but in the mean time, could you set “CPU_cores_to_use” to 1 instead of 0 (meaning all i.e. multi-core) and see if that still give you the same?

Cheers

Chris

[anciltech]

Just tried it with both 1 and 2 cores... 1 core works as expected, multiple tiles are created 2 cores causes the same issue as explained in my previous comment

It sounds like maybe the variable processed_list is cleared during/after multiprocessing?

[ChHarding]

Well, so can you work in single core mode for now? Or is that a show stopper for what you want to do?

I think the reason is 'TypeError("cannot pickle 'memoryview' object”)’ but I have to recreate that for myself to dig in deeper. I’m also thinking of re-vamping the way I use logging with a more flexible system (log guru) which supposedly can also log during multi processing (which the standard logger can’t …) so that should help in tracking down the issue.

On Oct 26, 2021, at 10:41, anciltech @.***> wrote:

Just tried it with both 1 and 2 cores... 1 core works as expected, multiple tiles are created 2 cores causes the same issue as explained in my previous comment

It sounds like maybe the variable processed_list is cleared during/after multiprocessing?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/ChHarding/TouchTerrain_for_CAGEO/issues/53#issuecomment-952066854, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEYDF5ICSUX2GOYB5VMFBLDUI3K33ANCNFSM5EJJFKZQ. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[anciltech]

Optimal would be a single tile that could be processed on multiple cores, as multiple tiles was more of a work around to begin with, both for performance and size constraints. But now that a single core can put together a large file, I can certainly utilize this to create the monster files I was hoping for. Only gains to be had are on performance, if the multicore functionality is able to be implemented in this way.