Exit during georeferencing

[KassieC]

I have a set of 126 primarily ortho photos that's part of a larger collection of 527 orthos and obliques that I'm trying to orthomosaic and running into problems. Things seem to progress quite nicely, until I hit the georeference step where it kicks up the following Traceback:

[WARNING] No coordinates file. Generating coordinates file: /var/www/data/3fcf93e8-40a7-4c46-b385-5f7569bb4e1c/odm_georeferencing/coords.txt

[DEBUG]   Line: WGS84 UTM 6N

Traceback (most recent call last):
  File "/code/run.py", line 46, in <module>
    plasm.execute(niter=1)
  File "/code/scripts/odm_georeferencing.py", line 164, in process
    geo_ref.parse_coordinate_system(tree.odm_georeferencing_coords)
  File "/code/opendm/types.py", line 293, in parse_coordinate_system
    self.utm_zone = int(ref[2][:len(ref) - 1])
ValueError: invalid literal for int() with base 10: '6N'

Full output is here. Data set of photos is here

I've blindly tried a few different options and everything seems to kick me out at this step.

Edit to add: I got this same result from a totally independent set of images. Here's the log file, and I'll link the (smaller) photo set when it uploads (Alaskan internet isn't great). Edit2: Added link to first data set; While I was waiting for the upload to finish I ran the Aukerman collection just fine.

[KassieC]

@smathermather -- I'm not quite sure I understand the issue. Are you saying that the problem is that the photos are centered on the nadir? I'm not sure how that'd be different from the AUkerman park example data... I've now run a 4th set that kicks me out at the generating coordinates file step, with the same traceback.

My python's rusty, but... it looks like it's expecting more characters in the UTM zone? I.e., UTM zone 10+. Or, that's how it looks... the 2nd to last line looks like it's just trying to grab the digits for the zone. Since the photos are all in zone 6 or 5 (for a different photoset not posted here) and not 10+, those all have got non-numeric data back because it's an unexpected length. (Forgive me if I'm wrong about this -I study squishy things and I'm reaching a ways back to classes I mostly slept through).

[dakotabenjamin]

I'm running the images now, first thing I did was delete DJI_0001 because of the sky in the image.

You may be on the right track, and may also be the farthest north anyone's ever tried to use ODM with before!

Check it out yourself:

ref = ['WGS84','UTM','6N']
ref[2]
Out[4]: '6N'
int(ref[2][:len(ref) - 1])
Traceback (most recent call last):
  File "C:\Users\dmb2\AppData\Local\Continuum\Anaconda2\lib\site-packages\IPython\core\interactiveshell.py", line 3066, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-5-744fd17e7387>", line 1, in <module>
    int(ref[2][:len(ref) - 1])
ValueError: invalid literal for int() with base 10: '6N'
ref = ['WGS84','UTM','61N']
int(ref[2][:len(ref) - 1])
Out[7]: 61

We need to change line 293 to say instead:

int(ref[2][:len(ref[2]) - 1])

Nice catch!

[smathermather]

Should we be using polar stereographic when we get too far north? UTM starts to fall apart in the polar and near polar regions.

[dakotabenjamin]

If you think that's a good idea and works in a general way for the latlong EXIF conversion, then let's do it, or find someone to champion.

Can we open an issue in OpenDroneMap/OpenDroneMap? This is not relevant to WebODM at this point.

[KassieC]

I'm running the images now, first thing I did was delete DJI_0001 because of the sky in the image.

Woops. Yeah, that shouldn't have been uploaded. I just batch-selected that flight and uploaded before I left it to its own devices.

You may be on the right track, and may also be the farthest north anyone's ever tried to use ODM with before!

There's more north than me! We've got datasets going from all over in AK for habitat mapping, rangeland management, that kinda stuff we're eager to try. In that vein,

Should we be using polar stereographic when we get too far north? UTM starts to fall apart in the polar and near polar regions.

I chatted a bit with my GIS specialist who I've been working with on some of these problems, and after the immediate reaction of "Why not Alaska Albers," and my pointing out that the world doesn't revolve around Alaska (I had the same reaction, so I have to cut him some slack!), he thinks that polar stereographic might be a reasonable. However, UTMs work reasonably well around here, and especially on the spatial scales a lot of folks work on for this sort of image acquisition. If you want to future-proof it for spatially large, yeah, absolutely, anything north (or south) of somewhere between 55 deg and 60 deg should seriously consider not working in UTMs. But in the short-term, it probably(?) would be OK.

Caveat: That's just our reaction after a talking about it for a while before the coffee kicked in. We (especially me) could be wildly ignorant about this.

We need to change line 293 to say instead: int(ref[2][:len(ref[2]) - 1])

I assume you made that edit? I'm a github neophyte and I'd be nervous about doing it myself. Also, there's the question of how to fix it on my box - I don't work much with docker, so I don't know how I'd force it to update ODM.

[dakotabenjamin]

I haven't yet, but it's a simple change so I'll have it up as a PR today. This will change the core ODM project, not WebODM, so once it's integrated there then you can run the upgrade using the standard process for WebODM.

[dakotabenjamin]

The changes have been added to odm core, we can close this once they've been reflected in node-odm

[eduard-kazakov]

Hi there! I have the same issue and found out that the problem is not really resolved with changing line to int(ref[2][:len(ref[2]) - 1])

I have error with zone 36N: invalid literal for int() with base 10: '36N'

It seemed like there are something like the newline character... So, i tried to modify types.py script and add some new DEBUG lines, and i got something interesting: look at length of input list log.ODM_DEBUG(len(ref))

3

look at length of element with UTM log.ODM_DEBUG(len(ref[2]))

4

!!!

look at what we try to convert to int log.ODM_DEBUG('raw: %s.' %ref[2][:len(ref[2])-1])

36N

wtf?

so, lets try to look at this list element, adding "!" to understand more log.ODM_DEBUG('raw: %s!' %ref[2][:len(ref[2])])

raw: 36N !

wow, as we can see the exclamation point is on new line, so we have something unnecessary here

I guess that it will be much correctly to use re.sub to extract digits instead of list substrings. self.utm_zone = int(re.sub("\D","",ref[2]))

I try it in my case and it works perfectly.

[KassieC]

Hi @silenteddie, hmm... interesting. I haven't been able to get it to work since the pull request but I assumed the changes needed to be reflected in node-ODM yet (though searching for the line I didn't find it - I assume this is because I'm looking in the 'wrong way'). I'm new to this whole Docker world; how are you getting to types.py? Dumb question, but I'd like to see if I can tinker with it myself (likely a dangerous thing, but I'm working in a scratch drive anyhow).

[pierotofy]

cc @pepperlk this looks like the same issue you reported on gitter.

Can confirm that a newline char will cause the problem to appear:

line = 'WGS84 UTM 17N\n'
ref = line.split(' ')
# match_wgs_utm = re.search('WGS84 UTM (\d{1,2})(N|S)', line, re.I)
if ref[0] == 'WGS84' and ref[1] == 'UTM':  # match_wgs_utm:
    datum = ref[0]
    utm_pole = ref[2][len(ref) - 1]
    utm_zone = int(ref[2][:len(ref[2]) - 1])
    print(utm_zone)
    print(utm_pole)
    # extract east and west offsets from second line.
    # We will assume the following format:
    # '440143 4588391'
    # update EPSG

I proposed a fix here https://github.com/OpenDroneMap/OpenDroneMap/pull/665

[DRIV3R78]

Hi there,

I´m new to ODM and using WebODM since yesterday. I made a set of about 192 photos, made with my Mavic Pro an the dronedeploy app. I have tried to different projects, but everytime it exits with ValueError: invalid literal for int() with base 10: '32N'

I´ve read thru this section but I don´t now, if there is a near solution, or if I have to do anything, to get rid of thist error.

The Photo Set can be found here. https://drive.google.com/open?id=0BxJQgbp3CSSHemhwUEszdnVGWG8

And the Task Output can be found attached. task_output.txt

Hope you can help. Best regards

Timo

[pierotofy]

@DRIV3R78 I've just pushed a patched image on docker. If you've installed WebODM using docker simply issue a docker pull opendronemap/node-opendronemap followed by a ./webdodm.sh down && ./webodm.sh start. If you used the Windows Installer and are using WebODM Manager, simply press the "Update" button.

[eduard-kazakov]

@TwoYaks @DRIV3R78 Or as temporary workaround you can in file /opendronemap directory/opendm/types.py replace string

self.utm_zone = int(ref[2][:len(ref[2]) - 1])

with

self.utm_zone = int(re.sub("\D","",ref[2]))

[DRIV3R78]

@pierotofy @TwoYaks Thanks for the quick reply. I´m using WebODM and did an update seconds ago. But now mit node is offline and I can´t process anything.

node-odm-1:3000 seems to be offline

Any further ideas?

[pierotofy]

@DRIV3R78 uups, made a small mistake when pushing the docker image. Try again the steps I proposed, should work now.

[DRIV3R78]

@pierotofy Thanks. Did the update. Now the node is back. I´m processing at the moment an will se, if the error is gone.

Will give an update here.

[DRIV3R78]

@pierotofy Works for me now. Thanks for the update. But one question. I have uploaded several images, but, the processed map don´t show the hole images. Can you have a lock.

Here are the log an the picture.

task_output.txt

And one further question. Is it possible with ODM to show an "Plant Health" Overlay or a "stockpile measure"?

Thanks allot

[pierotofy]

Try increasing the min-num-features parameter to attempt to fill the gaps when processing a task.

We are working toward adding plant health and stockpile measurements. See #53 and #241.

[KassieC]

@pierotofy I've tried to locate -i types.py and then 'grep' the relevant file (opendm/types.py; also tried it with some wildcards), and I don't see what I'm looking for. 'locate' returns too many hits to go through exhasutively, but just visually scanning over the directory placements, none of them are the file I'm looking for... how do I get into that file?

(Needless to say, the self.utm_zone = int(ref[2][:len(ref[2]) - 1]) fix didn't work for me either.)

[jollino]

I'm a complete newbie to ODM and I'm having the same issue; it's also the first time I use Docker so I'm not sure if/how I can actually change anything inside the image). In the version I'm using, the trackback already includes the patch (self.utm_zone = int(ref[2][:len(ref[2]) - 1])) but it still fails:

[DEBUG] running /code/build/bin/odm_georef -bundleFile /code/opensfm/bundle_r000.out -inputCoordFile /code/odm_georeferencing/coords.txt -inputFile /code/odm_texturing/odm_textured_model.obj -outputFile /code/odm_texturing/odm_textured_model_geo.obj -inputPointCloudFile /code/opensfm/depthmaps/merged.ply -outputPointCloudFile /code/odm_georeferencing/odm_georeferenced_model.ply -logFile /code/odm_georeferencing/odm_georeferencing_log.txt -outputTransformFile /code/odm_georeferencing/odm_georeferencing_transform.txt -georefFileOutputPath /code/odm_georeferencing/odm_georeferencing_model_geo.txt [DEBUG] Line: WGS84 UTM 33N

Traceback (most recent call last): File "/code/run.py", line 47, in plasm.execute(niter=1) File "/code/scripts/odm_georeferencing.py", line 164, in process geo_ref.parse_coordinate_system(tree.odm_georeferencing_coords) File "/code/opendm/types.py", line 318, in parse_coordinate_system self.utm_zone = int(ref[2][:len(ref[2]) - 1]) ValueError: invalid literal for int() with base 10: '33N'

These are images straight out of my Phantom 3 Advanced, which I can provide if needed.

Oddly enough, the same image set run via the latest WebODM does not trigger the same issue, though it takes a much longer time to be processed, as the depthmap is computer without it stating "with PATCH_MATCH". I'm not sure if that has got anything to do with it, though it seems completely unrelated.

[pierotofy]

The docker ODM image is synced to the latest master, which doesn't include https://github.com/OpenDroneMap/OpenDroneMap/pull/665. I manually patched the node-opendronemap docker image as a temporary fix for WebODM.

[dakotabenjamin]

I'm testing now. Should be merged within the hour.

[smathermather]

@DRIV3R78 -- you might try calibrating the Mavic Pro images to get better results. What do you think, @dakotabenjamin?

[hblanken]

We had the same issue at '56S' and we implemented both proposed fixes, reading and writing utm_zone. the fix propsed by @silenteddie and @pierotofy. Now it works. I suggest implementing it in master.

[KassieC]

I've pulled the latest, and re-run, etc. I'm still getting kicked out at that step:

Traceback (most recent call last): File "/code/run.py", line 47, in plasm.execute(niter=1) File "/code/scripts/odm_georeferencing.py", line 164, in process geo_ref.parse_coordinate_system(tree.odm_georeferencing_coords) File "/code/opendm/types.py", line 317, in parse_coordinate_system self.utm_pole = ref[2][len(ref) - 1] IndexError: string index out of range

I spent a bit reading up on how to get into the docker containers etc, and went in and edited the types.py file. It is definitely the most up-to-date version of the master, and a few edits later (and some copious cursing), I got it to get past that step - only to realize docker had filled my drive and I was out of space to keep trying. Unfortunately, docker neophyte I am, I threw the baby out with the bath water and I don't have the edited file anymore. I may have got a little over-ambitious with prune.

[pierotofy]

@KassieC did you do a "docker pull opendronemap/opendronemap"?

[KassieC]

@pierotofy Correct. Just to be sure...

Using default tag: latest latest: Pulling from opendronemap/opendronemap Digest: sha256:01c48a3da02907b5ba6a689605a6e2cec2bd2beac9aa3a39f0eec5a8d8912283 Status: Image is up to date for opendronemap/opendronemap:latest

After I './webodm.sh start' [...]

File "/code/opendm/types.py", line 317, in parse_coordinate_system self.utm_pole = ref[2][len(ref) - 1] IndexError: string index out of range

[pierotofy]

I see, try to run:

docker pull opendronemap/node-opendronemap
./webodm.sh down
./webodm.sh start

Does it work then?

[KassieC]

Using default tag: latest latest: Pulling from opendronemap/node-opendronemap Digest: sha256:ad40e5aa5642b80edfee8a04f1dd457f4ca767bce903a353f8c6c22567da0780 Status: Image is up to date for opendronemap/node-opendronemap:latest

I go ahead and run it after down/start, and...

Whole texturing procedure took: 67.117s [INFO] Running ODM Texturing Cell - Finished [INFO] Running ODM Georeferencing Cell [DEBUG] None [WARNING] No coordinates file. Generating coordinates file: /var/www/data/dd616f2e-2119-4878-9a73-695b43c49472/odm_georeferencing/coords.txt [INFO] Running georeferencing with generated coords file. [DEBUG] running /code/build/bin/odm_georef -bundleFile /var/www/data/dd616f2e-2119-4878-9a73-695b43c49472/opensfm/bundle_r000.out -inputCoordFile /var/www/data/dd616f2e-2119-4878-9a73-695b43c49472/odm_georeferencing/coords.txt -inputFile /var/www/data/dd616f2e-2119-4878-9a73-695b43c49472/odm_texturing/odm_textured_model.obj -outputFile /var/www/data/dd616f2e-2119-4878-9a73-695b43c49472/odm_texturing/odm_textured_model_geo.obj -inputPointCloudFile /var/www/data/dd616f2e-2119-4878-9a73-695b43c49472/opensfm/depthmaps/merged.ply -outputPointCloudFile /var/www/data/dd616f2e-2119-4878-9a73-695b43c49472/odm_georeferencing/odm_georeferenced_model.ply -logFile /var/www/data/dd616f2e-2119-4878-9a73-695b43c49472/odm_georeferencing/odm_georeferencing_log.txt -outputTransformFile /var/www/data/dd616f2e-2119-4878-9a73-695b43c49472/odm_georeferencing/odm_georeferencing_transform.txt -georefFileOutputPath /var/www/data/dd616f2e-2119-4878-9a73-695b43c49472/odm_georeferencing/odm_georeferencing_model_geo.txt

IndexError: string index out of range

[pierotofy]

This might be a different fault then; could you share your dataset so that we can try to replicate it?

[KassieC]

I've got the big data set above (in my original issue), but this one is a smaller one that gets to the same step before kicking out. There might be a few images you'll need to trim out, because when I originally uploaded this I forgot to remove some.

I'll add that as I mentioned last night, I edited types.py in the self.utm_pole = line to change (unsurprisingly) the indexing and got it working before I brilliantly rm'ed all my containers.

[KassieC]

@pierotofy Got around going through it again. If I edit self.utm_pole = ref[2][len(ref) - 1] to self.utm_pole = ref[2][len(ref[2]) - 1] it'll make it past that step, only to drop at a subsequent step:

[INFO] Will skip classification, only DSM is needed [DEBUG] running l2d_dems dsm /var/www/data/f499cd41-e456-43d7-b703-59f7ecc122d4/odm_dem --slope 10 --cellsize 2 -o -s /var/www/data/f499cd41-e456-43d7-b703-59f7ecc122d4/odm_dem/bounds.shp --outdir /var/www/data/f499cd41-e456-43d7-b703-59f7ecc122d4/odm_dem --maxsd 2.5 --maxangle 20 --resolution 0.04 --radius 0.04 0.12 0.36 1.08 --decimation 1 --gapfill

Error creating dsm bounds-0: Error creating dems: idw l2d_dems dsm completed (../var/www/data/f499cd41-e456-43d7-b703-59f7ecc122d4/odm_dem) in 0:02:00.106905 Traceback (most recent call last): File "/code/run.py", line 47, in plasm.execute(niter=1) File "/code/scripts/odm_dem.py", line 171, in process os.rename(os.path.join(odm_dem_root, 'bounds-0_dsm.idw.tif'), dsm_output_filename) OSError: [Errno 2] No such file or directory