Trying to use your toolchain

Hi!

I'm trying to use your toolchain to render some partial 360 panorama. The independant images are from my custom setup (v4mbike, v4mpack).

I had some difficulties to create the containers but after some trial/error I can open the opv-status web page. (I made some small commits on my fork https://github.com/Stefal/OPV_Ansible)

Now, i'm a little lost. I think that I need to import some images. Is there an example with a few images downloadable anywhere?

Thanks!

Hi @Stefal, Good job, for setting up the environment, next step is to upload data in your container. The easiest way is to use opv-status which is our backoffice (available on port 80), then you should access to the import page, you will must fill 3 field:

campaign name --> string
camera number --> int, you will have to adjust it with your hardware
campaign directory --> string, full path to your campaign dir, you can have example of campaign dir here and here, the campaign dir must be store in the container in the opv home /home/opv/

You may not want to use our backoffice, cause it isn't really flexible. You can directly use opv-import (which is run by opv-status) the readme is pretty well explained. In the readme you are looking for Make lot which build set of image and upload them to the container.

If you've got any trouble by importing data (or by doing any other thing), api logs are availabe here /home/opv/logs.

Next step is to stitch your panorama (I think you will have a lot of trouble here). opv-tasks do that for you, but be careful you will have to modify some tasks. To do this you will have to install opv-tasks in a develop mode:

(opv) opv@master:~$ pip uninstall opv-tasks (opv) opv@master:~$ cd dev/ (opv) opv@master:~/dev$ git clone https://github.com/OpenPathView/OPV_Tasks.git (opv) opv@master:~/dev$ cd OPV_Tasks (opv) opv@master:~/dev/OPV_Tasks$ python setup.py develop

Now you will be able to modify code in ~/dev/OPV_Tasks.

Thing I think you will need to change :

Disable rotate tasks
Change hugin configuration, here is a tutorial on how to generate a PTO file
Adapt or disable stichable task, if you disable it be careful to remove this condition in stitch task

Normally it's everything but I might forget one or two change. Now according to opv-tasks readme you can run :

opv-task makeall '{"id_lot": 130, "id_malette": 15 }' You should change id_lot and id_malette to your lot (you can get these information on opv-status by selecting a campaign).

Normally you could launch the stiching of all lot of a campaign by using spark tab but this will not work cause we do not use spark anymore and I didn't have the time to adapt opv-status.

If everything work you will be able to see the tiled panorama in opv-status.

I suggest you to take a look at our db api model to understand well our code. We have two other api (one for storing file, and one other to run opv-status.

If you need other information on how to use our environment or if you don't understand some feature, you can ask me question on this issue or directly on my twitter. Sorry for my broken english.

Thank you for all these informations @zyioump Now I need some time to dig into all of this.

We can continue in french if you want.

Hi @zyioump !

I'm trying to import your data, but it doesn't work. I don't know if I'm doing something wrong as I'm a newbie in linux, lxc, postgres and many other things :-)

I can't use the backoffice to import the images (errors) so here is what I've tried:

$ ssh root@opv_master
root@master:~# su opv

And there are some errors:

Error: EACCES: permission denied, scandir '/root'
(node:5460) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'get' of undefined
    at errorHandler (/home/opv/.nvm/versions/node/v11.2.0/lib/node_modules/npm/lib/utils/error-handler.js:205:18)
    at /home/opv/.nvm/versions/node/v11.2.0/lib/node_modules/npm/bin/npm-cli.js:78:20
    at cb (/home/opv/.nvm/versions/node/v11.2.0/lib/node_modules/npm/lib/npm.js:228:22)
    at /home/opv/.nvm/versions/node/v11.2.0/lib/node_modules/npm/lib/npm.js:266:24
    at /home/opv/.nvm/versions/node/v11.2.0/lib/node_modules/npm/lib/config/core.js:83:7
    at Array.forEach (<anonymous>)
    at /home/opv/.nvm/versions/node/v11.2.0/lib/node_modules/npm/lib/config/core.js:82:13
    at f (/home/opv/.nvm/versions/node/v11.2.0/lib/node_modules/npm/node_modules/once/once.js:25:25)
    at /home/opv/.nvm/versions/node/v11.2.0/lib/node_modules/npm/lib/config/core.js:112:20
(node:5460) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:5460) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
TypeError: Cannot read property 'loaded' of undefined
    at process.<anonymous> (/home/opv/.nvm/versions/node/v11.2.0/lib/node_modules/npm/lib/utils/error-handler.js:40:18)
    at process.emit (events.js:182:13)
/home/opv/.nvm/versions/node/v11.2.0/lib/node_modules/npm/lib/utils/error-handler.js:205
  if (npm.config.get('json')) {
                 ^

TypeError: Cannot read property 'get' of undefined
    at process.errorHandler (/home/opv/.nvm/versions/node/v11.2.0/lib/node_modules/npm/lib/utils/error-handler.js:205:18)
    at process.emit (events.js:182:13)
    at process._fatalException (internal/bootstrap/node.js:577:27)
nvm is not compatible with the npm config "prefix" option: currently set to ""
Run `npm config delete prefix` or `nvm use --delete-prefix v11.2.0 --silent` to unset it.

Let's continue

opv-make-lot /home/opv/test/SD --campaign-name=ma_campagne --number-of-devices=6

Some errors:

INFO:opv_import.controllers.cli.opv_make_lot:Make lot with parameters : {'cameras_dir': Path('/home/opv/test/SD'), 'csv_path': None, 'id_malette': 42, 'campaign_name': 'ma_campagne', 'number_of_devices': 6, 'campaign_desc': 'no description', 'api_uri': 'http://opv_master:5000', 'dm_uri': 'http://opv_master:5005', 'campaign_id_rederbro': 0, 'dm_file': None}
Traceback (most recent call last):
  File "/home/opv/venvs/opv/lib/python3.5/site-packages/urllib3/connection.py", line 159, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw)
  File "/home/opv/venvs/opv/lib/python3.5/site-packages/urllib3/util/connection.py", line 80, in create_connection
    raise err
  File "/home/opv/venvs/opv/lib/python3.5/site-packages/urllib3/util/connection.py", line 70, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/opv/venvs/opv/lib/python3.5/site-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/home/opv/venvs/opv/lib/python3.5/site-packages/urllib3/connectionpool.py", line 354, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python3.5/http/client.py", line 1106, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python3.5/http/client.py", line 1151, in _send_request
    self.endheaders(body)
  File "/usr/lib/python3.5/http/client.py", line 1102, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python3.5/http/client.py", line 934, in _send_output
    self.send(msg)
  File "/usr/lib/python3.5/http/client.py", line 877, in send
    self.connect()
  File "/home/opv/venvs/opv/lib/python3.5/site-packages/urllib3/connection.py", line 181, in connect
    conn = self._new_conn()
  File "/home/opv/venvs/opv/lib/python3.5/site-packages/urllib3/connection.py", line 168, in _new_conn
    self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f5c0006c668>: Failed to establish a new connection: [Errno 111] Connection refused

These errors seem related to socket.gethostbyname(socket.gethostname()) in ftp.py

There is something wrong with the hostname, but I don't know If I've done something wrong, or if there is a bug in the Ansible.

$ hostname
master
$ hostname -I
10.136.238.101 fdcf:d5b3:abbc:17c:216:3eff:feca:9615
$ hostname -i
hostname: Name or service not known
$ cat /etc/hostname
master

Hi @Stefal ! The nvm crash (first one) is due to su opv cause nvm was trying to read into /root but it was not allowed. It's not a problem but to avoid this crash you should use su - opv instead of su opv to change your current directory to opv home.

The opv-import error is a stranger one. Opv-import is trying to connect to the directory manager or the db api but it fail. So can you send me the result of this command netstat -nlpt(as root in the container), this command show all used port. You should check if

The db api is launched (port 5000)
The directory manager is launched (port 5005 and 5050) If one is missing you must restart the service :

systemctl restart opv-api systemctl restart opv-dm

If you restart the service and you can't even use the apis there are log in /home/opv/logs/ (please send me dm log and api log when you have a problem).

If the api are launched but it didn't work it may be a hosts problem. So can you ping opv_master (inside the container), if you can not please check this file /etc/hosts you should have this line 10.194.228.186 OPV_Master (with your own ip of course).

If the ping work does these curl work ?

curl http://opv_master:5000 #should return a 200 with a big json curl http://opv_master:5005 #should return a 404

Thanks, no error anymore with su - opv

Output of netstat -nlpt

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:2181            0.0.0.0:*               LISTEN      -               
tcp        0      0 10.136.238.101:2888     0.0.0.0:*               LISTEN      -               
tcp        0      0 0.0.0.0:5000            0.0.0.0:*               LISTEN      630/python3.5   
tcp        0      0 0.0.0.0:5001            0.0.0.0:*               LISTEN      379/python3.5   
tcp        0      0 10.136.238.101:3888     0.0.0.0:*               LISTEN      -               
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      -               
tcp        0      0 0.0.0.0:39251           0.0.0.0:*               LISTEN      -               
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      -               
tcp        0      0 0.0.0.0:5432            0.0.0.0:*               LISTEN      -               
tcp6       0      0 :::80                   :::*                    LISTEN      -               
tcp6       0      0 :::22                   :::*                    LISTEN      -               
tcp6       0      0 :::5432                 :::*                    LISTEN      -

The directory manager is missing.

I can't use systemctl restart as the ubuntu password for the ubuntu user is refused:

(opv) opv@master:~$ systemctl restart opv-dm
==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-units ===
Authentication is required to restart 'opv-dm.service'.
Authenticating as: Ubuntu (ubuntu)
Password: 
polkit-agent-helper-1: pam_authenticate failed: Authentication failure
==== AUTHENTICATION FAILED ===
Failed to restart opv-dm.service: Access denied
See system logs and 'systemctl status opv-dm.service' for details.

opv-api.log

INFO:dbrest.__main__:Starting OPV API using gunicorn with following arguments : {'--IDMalette': '42',
 '--db-location': 'postgres://opv:Testopv42@OPV_Master/opv',
 '--debug': 'False',
 '--help': False,
 '--port': '5000',
 '--process': '4',
 'run': True}
[2018-11-29 20:10:50 +0000] [625] [INFO] Starting gunicorn 19.9.0
[2018-11-29 20:10:50 +0000] [625] [INFO] Listening at: http://0.0.0.0:5000 (625)
[2018-11-29 20:10:50 +0000] [625] [INFO] Using worker: sync
[2018-11-29 20:10:50 +0000] [638] [INFO] Booting worker with pid: 638
[2018-11-29 20:10:50 +0000] [639] [INFO] Booting worker with pid: 639
[2018-11-29 20:10:50 +0000] [640] [INFO] Booting worker with pid: 640
[2018-11-29 20:10:50 +0000] [641] [INFO] Booting worker with pid: 641
INFO:dbrest.api:Starting dbrest API with IDMalette : '42'
INFO:dbrest.api:Starting dbrest API with IDMalette : '42'
INFO:dbrest.api:Starting dbrest API with IDMalette : '42'
INFO:dbrest.api:Starting dbrest API with IDMalette : '42'

opv-db.log

Traceback (most recent call last):
  File "/home/opv/venvs/opv/bin/opv_dm_web.py", line 22, in <module>
    from opv_directorymanager import Webservice, default_config
  File "/home/opv/venvs/opv/lib/python3.5/site-packages/opv_directorymanager/__init__.py", line 21, in <module>
    from opv_directorymanager.storage_service import *
  File "/home/opv/venvs/opv/lib/python3.5/site-packages/opv_directorymanager/storage_service/__init__.py", line 19, in <module>
    from opv_directorymanager.storage_service.ftp import FTP
  File "/home/opv/venvs/opv/lib/python3.5/site-packages/opv_directorymanager/storage_service/ftp.py", line 27, in <module>
    HOST = socket.gethostbyname(socket.gethostname())
socket.gaierror: [Errno -2] Name or service not known
Traceback (most recent call last):
  File "/home/opv/venvs/opv/bin/opv_dm_web.py", line 22, in <module>
    from opv_directorymanager import Webservice, default_config
  File "/home/opv/venvs/opv/lib/python3.5/site-packages/opv_directorymanager/__init__.py", line 21, in <module>
    from opv_directorymanager.storage_service import *
  File "/home/opv/venvs/opv/lib/python3.5/site-packages/opv_directorymanager/storage_service/__init__.py", line 19, in <module>
    from opv_directorymanager.storage_service.ftp import FTP
  File "/home/opv/venvs/opv/lib/python3.5/site-packages/opv_directorymanager/storage_service/ftp.py", line 27, in <module>
    HOST = socket.gethostbyname(socket.gethostname())
socket.gaierror: [Errno -2] Name or service not known
Traceback (most recent call last):
  File "/home/opv/venvs/opv/bin/opv_dm_web.py", line 22, in <module>
    from opv_directorymanager import Webservice, default_config
  File "/home/opv/venvs/opv/lib/python3.5/site-packages/opv_directorymanager/__init__.py", line 21, in <module>
    from opv_directorymanager.storage_service import *
  File "/home/opv/venvs/opv/lib/python3.5/site-packages/opv_directorymanager/storage_service/__init__.py", line 19, in <module>
    from opv_directorymanager.storage_service.ftp import FTP
  File "/home/opv/venvs/opv/lib/python3.5/site-packages/opv_directorymanager/storage_service/ftp.py", line 27, in <module>
    HOST = socket.gethostbyname(socket.gethostname())
socket.gaierror: [Errno -2] Name or service not known
Traceback (most recent call last):
  File "/home/opv/venvs/opv/bin/opv_dm_web.py", line 22, in <module>
    from opv_directorymanager import Webservice, default_config
  File "/home/opv/venvs/opv/lib/python3.5/site-packages/opv_directorymanager/__init__.py", line 21, in <module>
    from opv_directorymanager.storage_service import *
  File "/home/opv/venvs/opv/lib/python3.5/site-packages/opv_directorymanager/storage_service/__init__.py", line 19, in <module>
    from opv_directorymanager.storage_service.ftp import FTP
  File "/home/opv/venvs/opv/lib/python3.5/site-packages/opv_directorymanager/storage_service/ftp.py", line 27, in <module>
    HOST = socket.gethostbyname(socket.gethostname())
socket.gaierror: [Errno -2] Name or service not known
Traceback (most recent call last):
  File "/home/opv/venvs/opv/bin/opv_dm_web.py", line 22, in <module>
    from opv_directorymanager import Webservice, default_config
  File "/home/opv/venvs/opv/lib/python3.5/site-packages/opv_directorymanager/__init__.py", line 21, in <module>
    from opv_directorymanager.storage_service import *
  File "/home/opv/venvs/opv/lib/python3.5/site-packages/opv_directorymanager/storage_service/__init__.py", line 19, in <module>
    from opv_directorymanager.storage_service.ftp import FTP
  File "/home/opv/venvs/opv/lib/python3.5/site-packages/opv_directorymanager/storage_service/ftp.py", line 27, in <module>
    HOST = socket.gethostbyname(socket.gethostname())
socket.gaierror: [Errno -2] Name or service not known

I can successfuly ping opv_master from the master container.

/etc/hosts contains:

127.0.0.1 localhost

# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
10.136.238.101 OPV_Master
10.136.238.236 worker0

curl http://opv_master:5000 returns 404 with a json:

{
    "404": "The API call you tried to make was not defined. Here's a definition of the API to help you get going :)",
    "documentation": {
        "handlers": {
            "/sensors/{id_sensors}/{id_male ....

curl http://opv_master:5005 returns curl: (7) Failed to connect to opv_master port 5005: Connection refused

Don't forget the hostname problem:


Python 3.5.2 (default, Nov 12 2018, 13:43:14) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import socket
>>> socket.gethostname()
'master'
>>> socket.gethostbyname(socket.gethostname())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
socket.gaierror: [Errno -2] Name or service not known```

Ok, in /etc/hosts I had:

127.0.0.1 localhost

# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
10.136.238.101 OPV_Master
10.136.238.236 worker0

and in /etc/hostname: master

So inside it, I changed master to OPV_Master. Then I was able to import the data and seeing the campaign in opv-status.

Perhaps there is something to fix in the playbook.

Some news:

I had many errors with opv-tasks crashing with:

from opensfm import csfm
ImportError: No module named csfm

I tried to rebuild opensfm, but it was crashing with c++: internal compiler error: Killed (program cc1plus)

Everything was running in virtualbox. I've "upgraded" the guest to 2 cpu and 4GB of Ram, and Opensfm builded correctly.

Now, I see a Renault Scenic with a flat tire and a message on a wall, from somebody who is looking for love :smile:

Hi!

After many hours, I can stitch a complete campaign :-)

I have a few questions:

What is a "cp" ?
I had to edit the opv-status code because I have only 4 images per lot and CampaignInfoVue needed 6, a value stored in config.json. I was looking for a way to extract this information automatically, but it seems it's not stored with the lot or the campaign. However, if I'm right, opv-make-lot doesn't import incomplete lot. So the number of images should be a correct value. Right ?

When I launch a stitching process with celery, if I have more than 1 concurrency in my worker, and when I use the verdandi blender (and not enblend), the makeall task failed with this Traceback (partial)

self._session.cwd(path)
File "/home/opv/venvs/opv/lib/python3.5/site-packages/ftputil/error.py", line 138, in __exit__
raise TemporaryError(*exc_value.args, original_exception=exc_value)
ftputil.error.TemporaryError: 421 Control connection timed out.
Debugging info: ftputil 3.4, Python 3.5.2 (linux)

I get the same thing if I launch 2 opv-task makeall in parallel on opv_master.

I tried to create a path with opv-task pathfinder and the path editor in the front end, but I don't really understand the concept, and when I try opv-task webgen it crashes because of no usable lots.
Is there a way to delete a campaign from the database?
I'd like to have an OPV toolchain which could more easily accept others, "pano head". I think it's not an easy task as it needs some modifications in a lot of different part of the code. Did you already work on this ? I think will build a new V4MPOD with Gitup cams, and maybe you're working on a Rederbro v2 too ?

Welcome back @Stefal, I will try to answer you question :

A CP (in our database) is a table that store information about hugin toolchain. If I remember well CP mean control point, it's create when you launch cpfind and store these information.
OPV-import import incomplete lot, so the number of image isn't always 6. This allow us to compute the pourcent of incomplet lot and to see where and when we had trouble with our camera.
I think that the problem come from your computer, not from our toolchain. Hugin toolchain are really heavy and use a lot of ram. I think that when you launch two makeall in parallel it's crash because you don't have enough ram (maybe verdandi use more ram than enblend, I don't have the time to look at verdandi). You can verify that by launching 2 makeall in parallel and a dmesg -w, this command allow you to see kernel log. You are looking for an out of memory error.
To use path you must first create a path details (using the front end) from a campaign, then edit it. You will have to import panorama in this backoffice (they will be converted in path node). You can now launch graph reduction with the front end (you don't have to use the pathfinder task). It will reduce the number of path node and create path edge (path edge are link between path node). You can know save it and your path details is ready. You can create a virtual tour using your brandnew path details. Webgen tasks is no longer usable.
The api support delete request but you will have trouble to delete campaign cause you will first need to delete all campaign lot, all campaign sensors, all campaign panorama... So consider that you can't delete a campaign.
You are totally right. We will think about that.

OpenPathView / OPV_Ansible

Trying to use your toolchain #50