Support for multi-webmanager on the same machine

[gnthibault]

This PR is still a work in progress, but the basic set of features are needed are there and tested

[knro]

Why do you need multiple web managers on the same machine?

[gnthibault]

I am building a robotic observatory software platform, and while testing it with actual devices (not simulators) I realized that my device drivers were much much less stable than what I expected. I proceeded to setup separate indiserver/indiwebmanager pair for the problematic devices so that I could programmatically reset the server for the individual device that is causing issue without stopping other devices.

Simply stop the driver was not enough (as mentionned in issue https://github.com/knro/indiwebmanager/issues/55)

[knro]

I will investigate #55 and see if I can replicate it.

[gnthibault]

Ok thank you very much for investigating. Just to make it clear, this is mostly likely and indiserver or even a driver issue, not a indiwebmanager issue (I will close the indiwebmanager issue).

I still think it can be valuable to restart individual indiserver remotely through API in order to make startup sequence fully reproducible. Also I have been facing issues on my camera driver that only show up when they are not ran individually on their own indiserver. All in all after I have cleaned the PR I still think it would be valuable.

[gnthibault]

Ok, I am almost there, I added support to load remote driver from API as well

[gnthibault]

Ok no there are plenty of other errors:

Aug 21 22:01:56 robot-scope indi-web[1622]: File "/home/gnthibault/.local/lib/python3.10/site-packages/indiweb-0.1.7-py3.10.egg/indiweb/main.py", line 16, in Aug 21 22:01:56 robot-scope indi-web[1622]: from .indihub_agent import IndiHubAgent Aug 21 22:01:56 robot-scope indi-web[1622]: File "/home/gnthibault/.local/lib/python3.10/site-packages/indiweb-0.1.7-py3.10.egg/indiweb/indihub_agent.py", line 17, in Aug 21 22:01:56 robot-scope indi-web[1622]: os.makedirs(INDIHUB_AGENT_CONFIG) Aug 21 22:01:56 robot-scope indi-web[1622]: File "/usr/lib/python3.10/os.py", line 225, in makedirs Aug 21 22:01:56 robot-scope indi-web[1622]: mkdir(name, mode) Aug 21 22:01:56 robot-scope indi-web[1622]: FileExistsError: [Errno 17] File exists: '/home/gnthibault/.indihub'

Aug 22 23:17:59 robot-scope indi-web[4345]: Traceback (most recent call last): Aug 22 23:17:59 robot-scope indi-web[4345]: File "/home/gnthibault/.local/bin/indi-web", line 33, in Aug 22 23:17:59 robot-scope indi-web[4345]: sys.exit(load_entry_point('indiweb==0.1.7', 'console_scripts', 'indi-web')()) Aug 22 23:17:59 robot-scope indi-web[4345]: File "/home/gnthibault/.local/bin/indi-web", line 25, in importlib_load_entry_point Aug 22 23:17:59 robot-scope indi-web[4345]: return next(matches).load() Aug 22 23:17:59 robot-scope indi-web[4345]: File "/usr/lib/python3.10/importlib/metadata/init.py", line 171, in load Aug 22 23:17:59 robot-scope indi-web[4345]: module = import_module(match.group('module')) Aug 22 23:17:59 robot-scope indi-web[4345]: File "/usr/lib/python3.10/importlib/init.py", line 126, in import_module Aug 22 23:17:59 robot-scope indi-web[4345]: return _bootstrap._gcd_import(name[level:], package, level) Aug 22 23:17:59 robot-scope indi-web[4345]: File "", line 1050, in _gcd_import Aug 22 23:17:59 robot-scope indi-web[4345]: File "", line 1027, in _find_and_load Aug 22 23:17:59 robot-scope indi-web[4345]: File "", line 1006, in _find_and_load_unlocked Aug 22 23:17:59 robot-scope indi-web[4345]: File "", line 688, in _load_unlocked Aug 22 23:17:59 robot-scope indi-web[4345]: File "", line 883, in exec_module Aug 22 23:17:59 robot-scope indi-web[4345]: File "", line 241, in _call_with_frames_removed Aug 22 23:17:59 robot-scope indi-web[4345]: File "/home/gnthibault/.local/lib/python3.10/site-packages/indiweb-0.1.7-py3.10.egg/indiweb/main.py", line 79, in Aug 22 23:17:59 robot-scope indi-web[4345]: db = Database(db_path) Aug 22 23:17:59 robot-scope indi-web[4345]: File "/home/gnthibault/.local/lib/python3.10/site-packages/indiweb-0.1.7-py3.10.egg/indiweb/database.py", line 33, in init Aug 22 23:17:59 robot-scope indi-web[4345]: self.create(filename) Aug 22 23:17:59 robot-scope indi-web[4345]: File "/home/gnthibault/.local/lib/python3.10/site-packages/indiweb-0.1.7-py3.10.egg/indiweb/database.py", line 82, in create Aug 22 23:17:59 robot-scope indi-web[4345]: c.execute('UPDATE Version SET version=?', (version,)) Aug 22 23:17:59 robot-scope indi-web[4345]: sqlite3.OperationalError: database is locked Aug 22 23:17:59 robot-scope systemd[1]: indiwebmanager_guiding_camera.service: Main process exited, code=exited, status=1/FAILURE Aug 22 23:17:59 robot-scope systemd[1]: indiwebmanager_guiding_camera.service: Failed with result 'exit-code'.

[gnthibault]

Ok, finally fixed the nasty bug with remote driver stop mentionned here: https://github.com/indilib/indi/issues/1927 The whole indihub management still requires a somework unfortunately

[knro]

Thank you, can you please summarize the changes?

[gnthibault]

Thank you, can you please summarize the changes?

Yes, sure, as mentionned earlier, I am not yet done, but currently, these are the main changes:

• Replaced the OS-native process management to terminate indiserver (a brutal kill signal to any application called indiserver) with a proper python managed process, where only the indiserver launched by one indiwebmanager instance is killed upon server stop endpoint activation
• Changed (still need to clean up that part) the indiserver commandline so that a different indi FIFO file path and socket filepath is used per webmanager instance
• Addition of a start_remoter and stop_remote endpoints to enable dynamic loading/unloading of remote drivers through the API. This arguably could be removed, and treated inside existing start/stop_driver with an "if" based on a driver name regexp (containing an @). I am open to discussion here
• Commented-out (to be cleaned) the condition on remote driver stop that precluded the stop driver FIFO command from using the "-n" option, which caused the issue mentionned in https://github.com/indilib/indi/issues/1927

I still need to fix the indihub part, which might not be that easy, especially for indihub, because I don't know much about this topic yet.

[gnthibault]

Ok I am moving forward with indihub, which is actually even more simple to "pythonize" than the indi_server class. I still need to make sure there is no clash with the sqlite DB and do some additional tests, but I am getting there.

Now, I have a more general question for your @knro is there a chance that the PR will get accepted if the python2 compatibility is broken ? Thank you in advance for your help

[knro]

We don't need to support Python2. I didn't have time yet to look at this PR in detail, hopefully over this weekend. Sorry for the delay.

[gnthibault]

Ok thank you for your feedback. No problem for the review, you are actually doing so much for the community already ! There is still a bit of testing I'd like to do (for feature I am not personally using on a daily basis) + hardening the sqlite support. Hopefully PR will be really ready for review before this weekend

[gnthibault]

Ok @knro I think this is ready for review. Squashed all commits, removed too verbose logging + proper commit message.

[knro]

What's a good test scenario for this?

[gnthibault]

Good point, I will setup and describe a proper test with 2 servers running in parallel

[gnthibault]

I used the following lines to run two distinct instances of indiweb:

indi-web -v indi-web -v --fifo /tmp/indiFIFO_science_camera --indi-port 7626 --port 8625

Then go to your browser, (http://localhost:8624 and http://localhost:8625)and you should be able to see two separate instance of indiwebmanager, and play with them.

Thank you in advance for your review

[gnthibault]

Open for review, please let me know if there are any observations / suggestions ? @knro

[gnthibault]

Thank you so much @knro don't hesitate to ping me if there is a new issue related to this PR that I can help support or fix.