search

OpenDroneMap / ODM

A command line toolkit to generate maps, point clouds, 3D models and DEMs from drone, balloon or kite images. 📷
https://opendronemap.org
GNU Affero General Public License v3.0
4.9k stars 1.11k forks source link

Split-merge fails creating submodels in server #1010

Open x-ancin opened 5 years ago

x-ancin commented 5 years ago

How did you install OpenDroneMap? (Docker, natively, ...)?

Using Docker in a linux server

What's your browser and operating system? (Copy/paste the output of https://www.whatismybrowser.com/)

Command line via PuTTy

What is the problem?

When I run ODM in a local machine it works fine with and without splitting the dataset, using

docker run -it --rm -v G:/test:/datasets/code opendronemap/odm --project-path /datasets

or

docker run -it --rm -v G:/test:/datasets/code opendronemap/odm --project-path /datasets --split 10 --split-overlap 3

As long as I have the data in a local folder (either hard drive or USB external disk), it does everything as expected

But when I run it on the server, it works only without the split command:

docker run -it --rm -v /my-server/Project folder:/datasets/code opendronemap/odm --project-path /datasets Works fine, but as soon as I add --split 10 --split-overlap 3

to the code, I get the following error:

[INFO] running /code/SuperBuild/src/opensfm/bin/opensfm create_submodels >/var/www/data/44a86e01-7ff1-4848-a6b6-711097026c96/opensfm Traceback (most recent call last): File “/code/SuperBuild/src/opensfm/bin/opensfm”, line 34, in command.run(args) File “/code/SuperBuild/src/opensfm/opensfm/commands/create_submodels.py”, line 37, in run meta_data.load_clusters_with_neighbors()) File “/code/SuperBuild/src/opensfm/opensfm/large/metadataset.py”, line 154, in >create_submodels os.symlink(src_relpath, dst) OSError: [Errno 95] Operation not supported Traceback (most recent call last): File “/code/run.py”, line 56, in app.execute() File “/code/stages/odm_app.py”, line 93, in execute self.first_stage.run() File “/code/opendm/types.py”, line 376, in run self.next_stage.run(outputs) File “/code/opendm/types.py”, line 357, in run self.process(self.args, outputs) File “/code/stages/splitmerge.py”, line 65, in process octx.run(“create_submodels”) File “/code/opendm/osfm.py”, line 21, in run (context.opensfm_path, command, self.opensfm_project_path)) File “/code/opendm/system.py”, line 76, in run raise Exception(“Child returned {}”.format(retcode)) Exception: Child returned 1

It seems like it opensfm has problems read/writing the submodels folder. I am added to the docker user group, but I do not have sudo rights when I run the command.

What should be the expected behavior? If this is a feature request, please describe in detail the changes you think should be made to the code, citing files and lines where changes should be made, if possible.

The expected behavior is that ODM creates the submodel folders, and processes the dataset in chunks, so then I can extract each submodel's orthophoto and DSM to work with smaller tif files afterward

How can we reproduce this? (What steps did you do to trigger the problem? What parameters are you using for processing? If possible please include a copy of your dataset uploaded on Google Drive or Dropbox. Be detailed)

Run ODM with --split on a server disk

x-ancin commented 5 years ago

I have been going back and forth with the server admin, and I think we have found the problem: Symlinks has issues when running on a cifs file system, which requires adding the mfsymlink flag to the mount command, and that is why I was getting the error. After adding the mfsymlink flag, the process works fine. I am running a bigger dataset now to make sure, but it seems like it is fixed now.

smathermather commented 5 years ago

Yes, this makes perfect sense. The heavy use of symlinks is a definite gotcha for the file system.

Piero -- do you still consider this a bug, or should we request x-ancin adds something to the docs, if they can?

smathermather commented 4 years ago

I don't think this is a bug but a file system issue. Closing.

anakrino10 commented 1 month ago

I am running Win 11 and have the same problem.

A set of 1476 photos runs without issue, even with both feature quality and pc quality at high.

However, when set to split the dataset (tried 100, 300, 500 and 1000 with overlap = 0m) it is always failing after generating the submodel folders with no photos in it.

Can there be a similar issue on Windows where the ODMNode is not able to copy the photos over to the submodel folders?

If so, it is a bug as the user is not able to deduce from the task output what the real issue is.

Read out python error OSError: [WinError 1] Incorrect function: 'F:\WebODM\resources\app\apps\NodeODM\data\df3664fa-9528-4bce-9aa4-3941f10910f6\images\DJI_000A(4).JPG' -> 'F:\WebODM\resources\app\apps\NodeODM\data\df3664fa-9528-4bce-9aa4-3941f10910f6\opensfm\..\submodels\submodel_0000\images\DJI_000A(4).JPG'

Strange path string, why is "openfsm" and ".." part of the path string? Should the path string not be

'F:\WebODM\resources\app\apps\NodeODM\data\df3664fa-9528-4bce-9aa4-3941f10910f6\submodels\submodel_0000\images\DJI_000A(4).JPG'

without

\opensfm\..

smathermather commented 1 month ago

Related community forum post: https://community.opendronemap.org/t/webodm-submodel-creation-error-oserror-winerror-1-incorrect-function/22235

Confirmed: this is a file system issue. Perhaps we can do a file system check when using split-merge and warn the user. I wouldn't be against copying data on a file system that doesn't support symlinks, but that's probably a conversation.

I'll re-open, as it would be useful to at least warn the user, but I'll mark it as help wanted.