Is there any way to know how far through mapproxy-tiling is?

[jrjdavidson]

Just checking, as I've had the world dataset generating for a few days now. mapproxy-calippers tells me that the LOD range should go down to 17, but it seems to be working in the LOD 10-13, after a few days of processing.. Any thoughts? Just want to know if there is just too much data for the maching, and I should kill the process and think of another strategy.

See current log below:

2021-12-20 19:27:37 I3 (tiling) [2341(tile:13-5-1715)]: Processed tile 13-5-1715 (extents: -19988588.644686,3248267.954007:-19978804.705066,3258051.893627, srs: pseudomerc) [watertight subtree]. {tiling.cpp:process():329}
2021-12-20 19:27:37 I3 (tiling) [2341(tile:12-3-857)]: Processed tile 12-3-857 (extents: -19978804.705066,3248267.954007:-19959236.825825,3267835.833248, srs: pseudomerc) [watertight]. {tiling.cpp:process():339}
2021-12-20 19:27:37 I3 (tiling) [2341(tile:13-6-1714)]: Processed tile 13-6-1714 (extents: -19978804.705066,3258051.893627:-19969020.765445,3267835.833248, srs: pseudomerc) [watertight subtree]. {tiling.cpp:process():329}
2021-12-20 19:27:37 I3 (tiling) [2341(tile:13-7-1714)]: Processed tile 13-7-1714 (extents: -19969020.765445,3258051.893627:-19959236.825825,3267835.833248, srs: pseudomerc) [watertight subtree]. {tiling.cpp:process():329}
2021-12-20 19:27:37 I3 (tiling) [2341(tile:13-6-1715)]: Processed tile 13-6-1715 (extents: -19978804.705066,3248267.954007:-19969020.765445,3258051.893627, srs: pseudomerc) [watertight subtree]. {tiling.cpp:process():329}
2021-12-20 19:27:37 I3 (tiling) [2341(tile:13-7-1715)]: Processed tile 13-7-1715 (extents: -19969020.765445,3248267.954007:-19959236.825825,3258051.893627, srs: pseudomerc) [watertight subtree]. {tiling.cpp:process():329}
2021-12-20 19:32:39 I3 (tiling) [2341(tile:11-0-128)]: Processed tile 11-0-128 (extents: -20037508.342789,14988995.498610:-19998372.584307,15028131.257092, srs: pseudomerc) [watertight]. {tiling.cpp:process():339}
2021-12-20 19:32:39 I3 (tiling) [2341(tile:12-0-256)]: Processed tile 12-0-256 (extents: -20037508.342789,15008563.377851:-20017940.463548,15028131.257092, srs: pseudomerc) [watertight subtree]. {tiling.cpp:process():329}
2021-12-20 19:32:40 I3 (tiling) [2341(tile:12-1-256)]: Processed tile 12-1-256 (extents: -20017940.463548,15008563.377851:-19998372.584307,15028131.257092, srs: pseudomerc) [watertight subtree]. {tiling.cpp:process():329}
2021-12-20 19:32:41 I3 (tiling) [2341(tile:12-0-257)]: Processed tile 12-0-257 (extents: -20037508.342789,14988995.498610:-20017940.463548,15008563.377851, srs: pseudomerc) [watertight subtree]. {tiling.cpp:process():329}
2021-12-20 19:32:41 I3 (tiling) [2341(tile:12-1-257)]: Processed tile 12-1-257 (extents: -20017940.463548,14988995.498610:-19998372.584307,15008563.377851, srs: pseudomerc) [watertight subtree]. {tiling.cpp:process():329}
2021-12-20 19:32:41 I3 (tiling) [2341(tile:11-1-128)]: Processed tile 11-1-128 (extents: -19998372.584307,14988995.498610:-19959236.825825,15028131.257092, srs: pseudomerc) [watertight]. {tiling.cpp:process():339}
2021-12-20 19:32:41 I3 (tiling) [2341(tile:12-2-256)]: Processed tile 12-2-256 (extents: -19998372.584307,15008563.377851:-19978804.705066,15028131.257092, srs: pseudomerc) [watertight subtree]. {tiling.cpp:process():329}
2021-12-20 19:32:41 I3 (tiling) [2341(tile:12-3-256)]: Processed tile 12-3-256 (extents: -19978804.705066,15008563.377851:-19959236.825825,15028131.257092, srs: pseudomerc) [watertight subtree]. {tiling.cpp:process():329}
2021-12-20 19:32:41 I3 (tiling) [2341(tile:12-2-257)]: Processed tile 12-2-257 (extents: -19998372.584307,14988995.498610:-19978804.705066,15008563.377851, srs: pseudomerc) [watertight subtree]. {tiling.cpp:process():329}
2021-12-20 19:32:41 I3 (tiling) [2341(tile:12-3-257)]: Processed tile 12-3-257 (extents: -19978804.705066,14988995.498610:-19959236.825825,15008563.377851, srs: pseudomerc) [watertight subtree]. {tiling.cpp:process():329}
2021-12-20 19:33:17 I3 (tiling) [2341(tile:11-0-144)]: Processed tile 11-0-144 (extents: -20037508.342789,14362823.362898:-19998372.584307,14401959.121380, srs: pseudomerc) [watertight]. {tiling.cpp:process():339}
2021-12-20 19:33:18 I3 (tiling) [2341(tile:12-0-288)]: Processed tile 12-0-288 (extents: -20037508.342789,14382391.242139:-20017940.463548,14401959.121380, srs: pseudomerc) [watertight subtree]. {tiling.cpp:process():329}
2021-12-20 19:33:18 I3 (tiling) [2341(tile:12-1-288)]: Processed tile 12-1-288 (extents: -20017940.463548,14382391.242139:-19998372.584307,14401959.121380, srs: pseudomerc) [watertight subtree]. {tiling.cpp:process():329}
2021-12-20 19:33:19 I3 (tiling) [2341(tile:12-0-289)]: Processed tile 12-0-289 (extents: -20037508.342789,14362823.362898:-20017940.463548,14382391.242139, srs: pseudomerc) [watertight subtree]. {tiling.cpp:process():329}
2021-12-20 19:33:19 I3 (tiling) [2341(tile:12-1-289)]: Processed tile 12-1-289 (extents: -20017940.463548,14362823.362898:-19998372.584307,14382391.242139, srs: pseudomerc) [watertight subtree]. {tiling.cpp:process():329}
2021-12-20 19:33:19 I3 (tiling) [2341(tile:11-1-144)]: Processed tile 11-1-144 (extents: -19998372.584307,14362823.362898:-19959236.825825,14401959.121380, srs: pseudomerc) [watertight]. {tiling.cpp:process():339}
2021-12-20 19:33:19 I3 (tiling) [2341(tile:12-2-288)]: Processed tile 12-2-288 (extents: -19998372.584307,14382391.242139:-19978804.705066,14401959.121380, srs: pseudomerc) [watertight subtree]. {tiling.cpp:process():329}
2021-12-20 19:33:19 I3 (tiling) [2341(tile:12-3-288)]: Processed tile 12-3-288 (extents: -19978804.705066,14382391.242139:-19959236.825825,14401959.121380, srs: pseudomerc) [watertight subtree]. {tiling.cpp:process():329}
2021-12-20 19:33:19 I3 (tiling) [2341(tile:12-2-289)]: Processed tile 12-2-289 (extents: -19998372.584307,14362823.362898:-19978804.705066,14382391.242139, srs: pseudomerc) [watertight subtree]. {tiling.cpp:process():329}
2021-12-20 19:33:19 I3 (tiling) [2341(tile:12-3-289)]: Processed tile 12-3-289 (extents: -19978804.705066,14362823.362898:-19959236.825825,14382391.242139, srs: pseudomerc) [watertight subtree]. {tiling.cpp:process():329}
2021-12-20 19:34:07 I3 (tiling) [2341(tile:10-0-166)]: Processed tile 10-0-166 (extents: -20037508.342789,6966165.009798:-19959236.825825,7044436.526762, srs: pseudomerc) [watertight]. {tiling.cpp:process():339}
2021-12-20 19:36:27 I3 (tiling) [2341(tile:11-0-293)]: Processed tile 11-0-293 (extents: -20037508.342789,8531595.349078:-19998372.584307,8570731.107560, srs: pseudomerc) [watertight]. {tiling.cpp:process():339}
2021-12-20 19:36:51 I3 (tiling) [2341(tile:11-0-476)]: Processed tile 11-0-476 (extents: -20037508.342789,1369751.546870:-19998372.584307,1408887.305352, srs: pseudomerc) [watertight]. {tiling.cpp:process():339}
2021-12-20 19:38:20 I3 (tiling) [2341(tile:12-2047-485)]: Processed tile 12-2047-485 (extents: 20017940.463548,10527519.031661:20037508.342789,10547086.910902, srs: pseudomerc) [watertight]. {tiling.cpp:process():339}
2021-12-20 19:38:20 I3 (tiling) [2341(tile:13-4094-970)]: Processed tile 13-4094-970 (extents: 20017940.463548,10537302.971281:20027724.403168,10547086.910902, srs: pseudomerc) [watertight subtree]. {tiling.cpp:process():329}
2021-12-20 19:38:21 I3 (tiling) [2341(tile:13-4095-970)]: Processed tile 13-4095-970 (extents: 20027724.403168,10537302.971281:20037508.342789,10547086.910902, srs: pseudomerc) [watertight subtree]. {tiling.cpp:process():329}
2021-12-20 19:38:21 I3 (tiling) [2341(tile:13-4094-971)]: Processed tile 13-4094-971 (extents: 20017940.463548,10527519.031661:20027724.403168,10537302.971281, srs: pseudomerc) [watertight subtree]. {tiling.cpp:process():329}
2021-12-20 19:38:22 I3 (tiling) [2341(tile:13-4095-971)]: Processed tile 13-4095-971 (extents: 20027724.403168,10527519.031661:20037508.342789,10537302.971281, srs: pseudomerc) [watertight subtree]. {tiling.cpp:process():329}
2021-12-20 19:38:22 I3 (tiling) [2341(tile:11-1022-243)]: Processed tile 11-1022-243 (extents: 19959236.825825,10488383.273179:19998372.584307,10527519.031661, srs: pseudomerc) [watertight]. {tiling.cpp:process():339}
2021-12-20 19:38:22 I3 (tiling) [2341(tile:12-2044-486)]: Processed tile 12-2044-486 (extents: 19959236.825825,10507951.152420:19978804.705066,10527519.031661, srs: pseudomerc) [watertight]. {tiling.cpp:process():339}
2021-12-20 19:38:22 I3 (tiling) [2341(tile:13-4088-972)]: Processed tile 13-4088-972 (extents: 19959236.825825,10517735.092040:19969020.765445,10527519.031661, srs: pseudomerc) [watertight subtree]. {tiling.cpp:process():329}
2021-12-20 19:38:22 I3 (tiling) [2341(tile:13-4089-972)]: Processed tile 13-4089-972 (extents: 19969020.765445,10517735.092040:19978804.705066,10527519.031661, srs: pseudomerc) [watertight subtree]. {tiling.cpp:process():329}
2021-12-20 19:38:22 I3 (tiling) [2341(tile:13-4088-973)]: Processed tile 13-4088-973 (extents: 19959236.825825,10507951.152420:19969020.765445,10517735.092040, srs: pseudomerc) [watertight subtree]. {tiling.cpp:process():329}
2021-12-20 19:38:22 I3 (tiling) [2341(tile:13-4089-973)]: Processed tile 13-4089-973 (extents: 19969020.765445,10507951.152420:19978804.705066,10517735.092040, srs: pseudomerc) [watertight subtree]. {tiling.cpp:process():329}
2021-12-20 19:38:22 I3 (tiling) [2341(tile:12-2045-486)]: Processed tile 12-2045-486 (extents: 19978804.705066,10507951.152420:19998372.584307,10527519.031661, srs: pseudomerc) [watertight]. {tiling.cpp:process():339}
2021-12-20 19:38:22 I3 (tiling) [2341(tile:13-4090-972)]: Processed tile 13-4090-972 (extents: 19978804.705066,10517735.092040:19988588.644686,10527519.031661, srs: pseudomerc) [watertight subtree]. {tiling.cpp:process():329}
2021-12-20 19:38:22 I3 (tiling) [2341(tile:13-4091-972)]: Processed tile 13-4091-972 (extents: 19988588.644686,10517735.092040:19998372.584307,10527519.031661, srs: pseudomerc) [watertight subtree]. {tiling.cpp:process():329}
2021-12-20 19:38:22 I3 (tiling) [2341(tile:13-4090-973)]: Processed tile 13-4090-973 (extents: 19978804.705066,10507951.152420:19988588.644686,10517735.092040, srs: pseudomerc) [watertight subtree]. {tiling.cpp:process():329}
2021-12-20 19:38:22 I3 (tiling) [2341(tile:13-4091-973)]: Processed tile 13-4091-973 (extents: 19988588.644686,10507951.152420:19998372.584307,10517735.092040, srs: pseudomerc) [watertight subtree]. {tiling.cpp:process():329}
2021-12-20 19:38:22 I3 (tiling) [2341(tile:12-2044-487)]: Processed tile 12-2044-487 (extents: 19959236.825825,10488383.273179:19978804.705066,10507951.152420, srs: pseudomerc) [watertight]. {tiling.cpp:process():339}
2021-12-20 19:38:22 I3 (tiling) [2341(tile:13-4088-974)]: Processed tile 13-4088-974 (extents: 19959236.825825,10498167.212799:19969020.765445,10507951.152420, srs: pseudomerc) [watertight subtree]. {tiling.cpp:process():329}
2021-12-20 19:38:22 I3 (tiling) [2341(tile:13-4089-974)]: Processed tile 13-4089-974 (extents: 19969020.765445,10498167.212799:19978804.705066,10507951.152420, srs: pseudomerc) [watertight subtree]. {tiling.cpp:process():329}
2021-12-20 19:38:22 I3 (tiling) [2341(tile:13-4088-975)]: Processed tile 13-4088-975 (extents: 19959236.825825,10488383.273179:19969020.765445,10498167.212799, srs: pseudomerc) [watertight subtree]. {tiling.cpp:process():329}
2021-12-20 19:38:22 I3 (tiling) [2341(tile:13-4089-975)]: Processed tile 13-4089-975 (extents: 19969020.765445,10488383.273179:19978804.705066,10498167.212799, srs: pseudomerc) [watertight subtree]. {tiling.cpp:process():329}
2021-12-20 19:38:22 I3 (tiling) [2341(tile:12-2045-487)]: Processed tile 12-2045-487 (extents: 19978804.705066,10488383.273179:19998372.584307,10507951.152420, srs: pseudomerc) [watertight]. {tiling.cpp:process():339}
2021-12-20 19:38:22 I3 (tiling) [2341(tile:13-4090-974)]: Processed tile 13-4090-974 (extents: 19978804.705066,10498167.212799:19988588.644686,10507951.152420, srs: pseudomerc) [watertight subtree]. {tiling.cpp:process():329}
2021-12-20 19:38:22 I3 (tiling) [2341(tile:13-4091-974)]: Processed tile 13-4091-974 (extents: 19988588.644686,10498167.212799:19998372.584307,10507951.152420, srs: pseudomerc) [watertight subtree]. {tiling.cpp:process():329}
2021-12-20 19:38:23 I3 (tiling) [2341(tile:13-4090-975)]: Processed tile 13-4090-975 (extents: 19978804.705066,10488383.273179:19988588.644686,10498167.212799, srs: pseudomerc) [watertight subtree]. {tiling.cpp:process():329}
2021-12-20 19:38:23 I3 (tiling) [2341(tile:13-4091-975)]: Processed tile 13-4091-975 (extents: 19988588.644686,10488383.273179:19998372.584307,10498167.212799, srs: pseudomerc) [watertight subtree]. {tiling.cpp:process():329}

[vaclavblazek]

Hi, it's hard to predict. Maybe I can come up with some progress measuring. It's stopping at LOD 12 and 13 becaus it hits fully covered tile ("watertight") at full resolution so there is no change to have holes in more detailed tiles. Tiling complexity is given by the fractalness of the input dataset, i.e. how many details are there at the full resolution.

[jrjdavidson]

ok thanks. I guess 'watertight subtree' is one lod down(or up?) from a watertight tile.

The current machine seems to be CPU limited, I could potentially start a new VM with higher specs, or simply split the dataset into smaller more manageable chunks, but I have no idea whether the process is close to finishing (is at 50%? at 10% at 1%?), and so I'm hesitating between stopping or keeping the process going! Ah, the sunken cost fallacy..

[vaclavblazek]

Yes, watertight tile is just tile with no mask, or rather "all pixels valid" mask. Watertight subtree happens when full dataset resolution was used to generate the mask and thus all tile's children are watertight as well so we can skip all those tiles.

Just FYI, tiling processes dataset in reference frame's tile tree and marks individual tiles as "present"/"missing" and "present" tiles are further divided into two categories: "has holes" and "watertight". This info is 1) directly used as tileset's tileindex (used in various backend parts, like data merging), 2) generating tileset's metatiles and 3) referenced when generating/serving tiles (missing: no tile, watertight: mesh with implicit all-valid mask, non-watertight: mesh + validity mask).

[jrjdavidson]

An update: the dataset finished overnight! yay!

Also- not sure where to put this but our MOOC that uses vts won a prize, I'd like to think that my work (and your help!) with vts has contributed to that!

[vaclavblazek]

Wow, great news! Congrats!