More changes at the USGS

[mathuin]

The USGS folks have changed their product keys. I need to learn the new ones. They've also changed how download options work, which changes how downloading works, and they have not yet documented the changes.

I have submitted a support request and will update this ticket as things happen.

[mathuin]

Okay, called the USGS, they asked me to send an email, I sent an email. While working on the email, I figured out that they haven't actually changed the product IDs so much as blanked out the ones I use (L01 L06 NED ND3) as the titles are still there in the output, just no actual product IDs. They still work by hand in download options, but haven't tried to really download them yet. Not sure if it's accidental or on purpose or what.

[MinorMajor]

I was disappointed to find that after spending several hours acquiring/installing Python and the many dependencies for TopoMC that GetRegion.py failed with "AttributeError: No products are available for this location!". Then upon finding/reading your latest post at minecraftforum.net about this latest USGS-related bug, I was pretty PO'd.

But, I understand the difficulties in referencing outside sources and can imagine that it's even worse when dealing with government agencies that have no concept of "unintended consequences".

I'm anxious to generate my first map and then some as my goal is to create a 1:1 of portions of Florida. I'm hoping to simulate the experience of walking the terrain as research for a fictional story. I've been struggling to find/create greyscale elevation images to import into WorldPainter, as the resolution levels aren't fine enough.

I kept notes on the installation to my iMac running OS X 10.8 "Mountain Lion", and am willing to draft a how-to for your audience.

I think that your mention of offering a web service for delivering TopoMC output is an excellent idea. The demand would be great and I know that I would be willing to pay for it. The ideal service would accept the requester's parameters and gen the map for download ...but the hosting server would likely need to be a very high-spec box. Alternatively, your TopoMC server could just offer organized libraries of maps gen'd at leisure on other hosts (yours or contributors with functional TopoMC installations).

Keep up your efforts! Again, I think this is a great tool whose results will appeal to a lot of Minecrafter's. Just look at the enthusiasm for WesterosCraft (http://westeroscraft.com/) and its stunning collaborative art.

Cheers.

[mathuin]

Thanks for the support! I'm using the downtime productively and making major changes to the software, ranging from improved OpenCL support (half-done) to PEP8 compliance (mostly done) and turning TopoMC into a buildable package (just begun).

Other than the USGS failures, the number one complaint I've received is that the software only works for the US. I've downloaded some test data for global elevation and landcover and am looking at writing a version which will use that downloaded test data. If I can validate this approach, I may be able to get out from under the USGS's thumb and continue using high-resolution data for the US by sending a hard drive to the USGS for them to fill up with terabytes of data which I can then use locally. Once I figure out how to do that well, I can look at putting TopoMC in the cloud, running on something like AWS, if I can figure out a decent way to handle billing.

Anyway. Keep taking notes on your OS X installation -- bonus points for OpenCL support -- and when you're done you can add them to the wiki. If you need special permission, let me know!

[mathuin]

It's been two weeks, and still no word from the USGS. So much for releasing 1.0 next week on the two-year anniversary of the project. Whee! School has started up so I don't have as much time as I'd like, but I have started looking at other data sources as I mentioned above. It's a pain to redo a lot of the numerical craziness, so if someone else wants to jump into that stuff, please do. I think I'm going to focus on the packaging aspect for now, at least until my daily "does the USGS stuff work again" check passes with flying colors.

[mathuin]

Yep, another week, another lack of response from the USGS. I emailed them again. :-( Sorry, folks.

[mathuin]

Yet another week. Sometime early next week, I'm calling them again. Again.

[mathuin]

Ugh. I now know why they haven't been responding to my emails or phone calls. They shut down the services upon which TopoMC depends as of mid-December.

http://nationalmap.gov/data_delivery/ned.html

http://nationalmap.gov/data_delivery/nlcd.html

So now it looks like I need to make some serious changes. Ugh.

[mathuin]

So I finally got a response from the USGS. They did discontinue the products I was using. They do have them available as tiles. They do have a different method for retrieving them. It should still be doable from the command line. So major architectural changes may not be necessary after all. Stay tuned.

[bidwbb]

Like MajorMinor, I managed (actually, my stepson managed) to install all the dependencies and ran into the same issue and then, thanks to the error message, found this thread. TopoMC is so cool. I hope you get it working again. In the meantime, there is one particular region north of Boston I'd really like to get voxelized for a Minecraft world. Any suggestions? Any workaround?

Thank you!!

[mathuin]

There is a map of the greater Boston area available on the showcase which may suit your needs. If not, you'll have to wait until I fix TopoMC.

Speaking of which: this week is finals week. Next week is spring break. I plan on spending most of my break on my open source projects, with TopoMC at the top of the list! My goal is to hotfix the master branch with working code, then port those changes into the development branch and feature branches. Once that's done, I can finish the feature branches (stuff like making the code package-friendly) and update develop and then spin a new release. Finally. :-)

Thanks for your patience!

Jack.

[bidwbb]

Jack, how wonderful to get a response from you, thanks!

Here's what my dream is: to get a "basemap", from TopoMC or otherwise, of the Breakheart Reservation north of Boston. On June 2nd, I'm directing an orienteering meet there, and I'd like to turn it into a team event where teams visit as many checkpoints as possible in a fixed time, 2 or 3 hours. I thought it would be neat if we could get a Minecraft game up and running at least a month before that, with the topography matching Breakheart as much as possible. I'm not a Minecraft person, but my son does Minecraft. Here's a link to a short demo http://www.youtube.com/watch?v=WlA1xj06K3k he made in "vanilla" Minecraft for an orienteering course. I've seen videos of survival games that have been built in Minecraft, where pairs of players start off working together. I picture something like that for the orienteering version. Here's a link to the orienteering maphttp://www.sprintseries.org/doma/show_map.php?user=ali&map=977. Once David gets an MC base map, he could probably use MCEdit himself to decorate it with appropriate vegetation, trails, boulders, lakes and whatnot. I'm not sure about whether he could do the mods that we'd want to create the orienteering game.

I fantasized about having the winners of the online game get a free trip to Boston to compete on the real terrain, for real...

Anyway, I'm not sure what resolution would be needed or would be possible from TopoMC. The other thing I thought about doing was getting LIDAR data and somehow transforming it to voxels. I know people who create base maps for orienteering maps from LIDAR data, and they have the Boston area LIDAR. I guess I thought of trying to get as fine scale as one-meter voxels,

Thanks

Barb Bryant

On Mon, Mar 18, 2013 at 4:45 PM, Jack Twilley notifications@github.comwrote:

There is a map of the greater Boston area available on the showcase which may suit your needs. If not, you'll have to wait until I fix TopoMC.

Speaking of which: this week is finals week. Next week is spring break. I plan on spending most of my break on my open source projects, with TopoMC at the top of the list! My goal is to hotfix the master branch with working code, then port those changes into the development branch and feature branches. Once that's done, I can finish the feature branches (stuff like making the code package-friendly) and update develop and then spin a new release. Finally. :-)

Thanks for your patience!

Jack.

— Reply to this email directly or view it on GitHubhttps://github.com/mathuin/TopoMC/issues/25#issuecomment-15080636 .

[mathuin]

I looked at the Breakheart Reservation, and it's probably outside the existing map, so the showcase won't be very helpful for you.

TopoMC only supports USGS data. The elevation data the USGS has for the reservation is 1/9-arc-second\ resolution or about three meters. I think the software would do a good job generating a 1:1 map from that data, given a sufficiently powerful computer (mine probably couldn't make one that big at 1:1) and I imagine your son would be able to polish any rough edges.

One important note on that 1/9-arc-second data. Right now it's available in the old format, unlike 1/3-arc-second and 1-arc-second which are only available in the new format. 1/9-arc-second will be available in the new format in April. The rewrite I will be doing next week will be for the new format, so 1/9-arc-second data will not be usable until sometime in April. The 1/3-arc-second data would let you two make test maps to make sure you're getting what you expect.

Jack.

[bidwbb]

Sweet, thanks!

On Tue, Mar 19, 2013 at 12:21 PM, Jack Twilley notifications@github.comwrote:

I looked at the Breakheart Reservation, and it's probably outside the existing map, so the showcase won't be very helpful for you.

TopoMC only supports USGS data. The elevation data the USGS has for the reservation is 1/9-arc-second\ resolution or about three meters. I think the software would do a good job generating a 1:1 map from that data, given a sufficiently powerful computer (mine probably couldn't make one that big at 1:1) and I imagine your son would be able to polish any rough edges.

One important note on that 1/9-arc-second data. Right now it's available in the old format, unlike 1/3-arc-second and 1-arc-second which are only available in the new format. 1/9-arc-second will be available in the new format in April. The rewrite I will be doing next week will be for the new format, so 1/9-arc-second data will not be usable until sometime in April. The 1/3-arc-second data would let you two make test maps to make sure you're getting what you expect.

Jack.

— Reply to this email directly or view it on GitHubhttps://github.com/mathuin/TopoMC/issues/25#issuecomment-15125188 .

[KermMartian]

I'm disappointed to find this thread; sorry to hear that the USGS is making your life difficult. I'm working on a research project on large=scale sparse arrays, and I wanted to use generation of real-world-based Minecraft maps as a fun demo application. Your work looked perfect to get an idea of what data the USGS has available for automated access. I hope that everything with the re-write works smoothly, and I look forward to helping test the code.

Edit: By the way, given that this is Minecraft and all, do you know of anything topological data source that provides 1-meter resolution? I'm hoping to use New York City for my test, since I live here and its complexity would adequately stress my system.

[mathuin]

I don't know of any 1-meter sources for NYC itself, but the USGS has the 1/9-arc-second data I mentioned above for the city and it's three meters which isn't exactly shabby. http://gis.ny.gov/elevation/dem-status.htm shows some 2-meter data and I thought I saw a mention on that site of 1-meter shoreline data.

I am making significant progress today with TopoMC. I am now able to request and download tiles from the command line based on coordinates. The tiles are 1x1 degree (for elevation) and 3x3 degree (for landcover) in size so I'm caching downloaded files to save bandwidth. It's frustrating that the typical user will have to download hundreds of megabytes of data, the vast majority of which they will not need for their small regions. The least I can do is keep them from having to do it twice. :-)

I expect to roll out a hotfix to the master branch sometime this evening which will make TopoMC work as it did before. I will then apply the hotfix to the develop branch so I can go back to working on new features, like installing TopoMC as a package, changing how data is stored, and more. Stay tuned -- the best is yet to come!

[mathuin]

Okay, I have code that can now download tiled files from the USGS and generate Minecraft worlds. It is extremely inefficient. Once I fix the major problems, I'll merge the hotfix into master. Just wanted to let you guys know I am making progress. :-)

[bidwbb]

Thank you!

On Wed, Mar 27, 2013 at 2:15 AM, Jack Twilley notifications@github.comwrote:

Okay, I have code that can now download tiled files from the USGS and generate Minecraft worlds. It is extremely inefficient. Once I fix the major problems, I'll merge the hotfix into master. Just wanted to let you guys know I am making progress. :-)

— Reply to this email directly or view it on GitHubhttps://github.com/mathuin/TopoMC/issues/25#issuecomment-15505958 .