Browser Access to USB Drive and Hotspot Setup

[HxPI]

Please provide support to remedy issues with 403 errors related to USB drive access. Thanks.

[aaronwmorris]

If you could run the following commands and provide the output. I think the problem is the permissions on /media/astroberry

ls -ld /media
ls -ld /media/astroberry
ls -ld /media/astroberry/ALLSKY

[HxPI]

drwxr-xr-x 3 root root 4096 Jun 9 2020 /media drwxr-x—-+ 3 root root 4096 Feb 22 18:31 /media/astroberry drwxr-xr-x 9 astroberry astroberry 65536 Dec 31 1969 /media/astroberry/ALLSKY

[aaronwmorris]

I am investigating how Astroberry/Linux decides what permissions to set.

[aaronwmorris]

Try adding the www-data user to the admin group and restarting nginx

sudo usermod -a -G admin www-data

sudo systemctl restart nginx

I am not really keen on using this method, but from what I am reading, the admin group has permissions to access storage devices.

[HxPI]

Received message "usermod: group 'admin' does not exist"

[HxPI]

There is an "adm" group according to groups command. Perhaps that's it?

[aaronwmorris]

You are right, try adm

[HxPI]

Still 403 Forbidden. Will try reboot.

[aaronwmorris]

You may also try adding to the plugdev group and restart nginx.

sudo usermod -a -G plugdev www-data

[HxPI]

Same 403 forbidden

[aaronwmorris]

Okay, I am working remotely on my test astroberry system so I cannot plug in a USB stick right now. Tomorrow, I can explore this in more detail when I am next to my raspberry pi. I'll figure out the way to get this working.

[HxPI]

Sounds good. Thanks.

[aaronwmorris]

Well, I feel kind of silly now. I do not think this was due to any default mount options. The /media/astroberry folder is just a static folder.

Adding the execute bit for other on the /media/astroberry folder should be good enough to fix this.

sudo chmod o+x /media/astroberry

[HxPI]

Everything looks good now except Latest tab is looking for latest.jpg and I have the image format set to png so the file should be latest.png?

[HxPI]

Well now I have an issue with the astroberry wifi hotspot. I am unable to connect to it. Any suggestions how to fix that? Otherwise will need to reinstall astroberry, then indi-allksy, then set /media/astroberry permissions I suppose. I'm using the ethernet port for testing.

[HxPI]

Now SQM tab shows nothing! It used to show an image but now nothing and web developer tools shows no errors, just loading next images.

[HxPI]

Also the last image is way behind the current time, like 8 minutes and seems to fall further behind. Is this normal or is the RPi3 with Astroberry too slow to handle this?

[HxPI]

So a reset brought the SQM back but there seems to be an issue with large sudden changes in brightness. When I turned the lights off, the charts stopped receiving updates and the chart data was alternating display data points order low/high and high/low. The SQM tab says problem loading page.

[HxPI]

I'm going to say there is still more work to do with getting this to work with astroberry. I'll try the rasbian load and see if it fairs better. Otherwise, I'll just keep astroberry and fire capture going until this all gets sorted out. Best wishes and I look forward to seeing this working well once all the bugs are sorted out. Thanks again.

[aaronwmorris]

I have merged PR #95 so that the latest image is correctly loaded based on the image type selected in the config.

Running on Astroberry with a Rasberry PI 3 is likely going to be problematic due to the constrained memory. You could get around some of this by disabling the GUI, but there are still a lot of background services running on Astroberry.

Building the timelapse video with ffmpeg is also a bit of a memory hog. Building a 1K video usually requires about 512MB of available memory. Even my Raspberry PI 3 cameras occasionally run out of memory doing this on vanilla Raspbian installs.

It really is better to run on a plain Raspbian (non-desktop) minimal install.

Disable GUI

sudo systemctl set-default multi-user.target
# reboot

Re-enable GUI

sudo systemctl set-default graphical.target
# reboot

[aaronwmorris]

I really appreciate the feedback. All of your suggestions and findings have been very helpful.

[HxPI]

No problem. I’m loading Rasbian 10 32bit and will install indi-allsky to see if it will work with the USB drive.

[aaronwmorris]

I am hunting down any other bugs there may be with the PNG setting. I merged #96 so that star trails and keograms are generated with the correct image type.

[HxPI]

With Raspberry Pi OS installed, how is the hotspot setup? I don’t see a hotspot and cannot connect to indi-allsky without RPi hotspot. The hotspot was already setup with astroberry.

[aaronwmorris]

You just know how to find gaps. :-)

I have not yet had a need to setup a hotspot, but I can work on that next.

[HxPI]

Yeah hopefully that’s the last one. Wi-Fi access will allow VNC and Web interface access.

[aaronwmorris]

I have completed my hotspot setup script if you would like to try it. It is at misc/setup_hotspot.sh.

I have tested it on Raspbian 10 and 11.

[HxPI]

I did apt update/upgrade/dist-upgrade, then git clone indi-allsky. I am now getting an error when running ./setup.sh:

###############################################

Welcome to the indi-allsky setup script

###############################################

Distribution: Raspbian Release: 11 Arch: armv7l

INDI_DRIVER_PATH: /usr/bin INDISERVER_SERVICE_NAME: indiserver ALLSKY_SERVICE_NAME: indi-allsky GUNICORN_SERVICE_NAME: gunicorn-indi-allsky ALLSKY_ETC: /etc/indi-allsky HTDOCS_FOLDER: /var/www/html/allsky DB_FOLDER: /var/lib/indi-allsky INSTALL_INDI: true

Setup proceeding in 10 seconds... (control-c to cancel)

Fixing git checkout ownership Installing packages...

There are not prebuilt indi packages for this distribution Please run ./build_indi.sh before running setup.sh

Unable to continue with indi-allsky setup. Please advise.

[HxPI]

I didn’t want Raspbian 11 but somehow apparently it got installed. So I’m using another script which is building the indi packages. It also setup a hotspot so that may be good enough. Install and testing in progress.

[HxPI]

indi packages are built. Now getting this error when running ./setup.sh

E: Unable to locate package indi-full E: Unable to locate package indi-rpicam

I wonder if this has something to do with repositories?

Hotspot is working but nothing comes up for https://raspberrypi.local. I guess indi-allsky didn't finish install. indiserver is installed though

2022-03-12T10:43:22: startup: indiserver Usage: indiserver [options] driver [driver ...] Purpose: server for local and remote INDI drivers INDI Library: 1.8.8 Code 1.8.8-tgz. Protocol 1.7. Options: -l d : log driver messages to /YYYY-MM-DD.islog -m m : kill client if gets more than this many MB behind, default 128 -d m : drop streaming blobs if client gets more than this many MB behind, default 5. 0 to disable -p p : alternate IP port, default 7624 -r r : maximum driver restarts on error, default 10 -f path : Path to fifo for dynamic startup and shutdown of drivers. -v : show key events, no traffic -vv : -v + key message content -vvv : -vv + complete xml driver : executable or [device]@host[:port]

Not sure what is going on. Install needs to be easier than this!

[HxPI]

Reinstalled Raspbian 10 32bit and indi-allsky. After running hotspot setup, my iPad will not connect to the hotspot. It says the weak security iand just sits there spinning. How do I access the hotspot settings on Raspbian? Is there any way to set the security to at least WPA2?

[HxPI]

Was able to install Network Manager and verify hotspot settings. Able to connect to hotspot now, for some unknown reason it just started working.

There is an issue with images not saving in PNG. I have FITs saved because RAW was also selected. Latest image is in PNG but no other images. Of course all the tabs that depend on the PNG are not working either.

[HxPI]

Changed to JPEG no difference. The latest image seems to be working though and does save in the selected format and is kept up to date. However, there are no JPEG or PNG images being saved to the applicable folder. I already set the execute permission on /media/pi. However, that folder won't let me change the settings to allow anyone to access. Probably not the issue if latest image is being saved there.

It would be most helpful to have the option of an image file to download onto the microSD with instructions how to customize network and security settings.

[aaronwmorris]

indi packages are built. Now getting this error when running ./setup.sh

E: Unable to locate package indi-full E: Unable to locate package indi-rpicam ... Not sure what is going on. Install needs to be easier than this!

I think what happened here is your system was detected as Debian 11 instead of Raspbian 11 after you ran the build script for INDI. There was a bit of code left over in the Debian 11 section that tried to install indi packages that should not have been there. I fixed that.

Something in the package dependencies used to compile INDI is causing the system to suddenly report as Debian 11. I have seen this before, but I do not know what to do to fix this. The Debian 11 detection also causes issues with the python modules since there are no pre-built python wheels, the python modules with C dependencies have to be built from source.

Trying to "do the right thing" is a challenge when supporting multiple distributions and multiple architectures with custom software that has to be built from source.

Changed to JPEG no difference. The latest image seems to be working though and does save in the selected format and is kept up to date. However, there are no JPEG or PNG images being saved to the applicable folder. I already set the execute permission on /media/pi. However, that folder won't let me change the settings to allow anyone to access. Probably not the issue if latest image is being saved there.

I am not sure what is happening with your images. I am currently chasing down a bug in the PNG code where it is taking up to 30 seconds to compress a PNG image. You might be hitting this issue. Hopefully the indi-allsky log can show you error messages.

https://github.com/aaronwmorris/indi-allsky/wiki/PNG-Performance

However, my test code at testing/imageCompressTest.py always shows sub-second compression times for PNG.

In the meantime, I would recommend sticking with JPEG.

[HxPI]

My RPi3 keeps crashing so this is not going to work for what I want. I ordered a RPi4 4GB and will try that. What OS do you recommend for running indi-allsky on the RPi4? Also, can you verify indi-allsky USB drive storage will work on said OS without permissions issues? I prefer PNG or TIFF for maximum image quality. I don’t even want it compressed, if that can be an option. I could also use FITs but this adds an extra step to get debayered image. Thanks.

[aaronwmorris]

Raspbian 10 (Legacy) still has the easiest experience.

All of my permanent camera installations are RPi3s and I have trouble with them crashing from time to time. I implemented a watchdog so they automatically reboot if they lock up. Ever since implementing the watchdog I have not had to manually intervene. https://github.com/aaronwmorris/indi-allsky/wiki/Raspberry-Pi-Watchdog

I will go investigate getting the automount support setup for the USB drives. I am also still looking into the PNG issue.

[HxPI]

Can indi-allsky be setup on Ubuntu Mate?

[aaronwmorris]

I will go download that and see if there are any necessary modifications to the setup script.

[aaronwmorris]

I have merged PR #118 for the Ubuntu MATE RPi support. You will have to run the ./misc/build_indi.sh script to build INDI from source since there are no binaries available.

If it is Ubuntu MATE on x86_64 (intel) then that should work with the existing setup.

[HxPI]

For dark frames, are they created and/or applied depending on temperature? For example, operating in 32F vs 80F will be a large difference in camera temp and resulting dark frame. Seems there should be various dark frames with filename indicating applicable temperature and applied depending on camera temperature. Perhaps every 5 degrees is sufficient?

[gilesco]

Not too many people have a temperature controlled camera as their allsky camera, so capturing dark frames at various temperatures would be difficult to achieve.

[HxPI]

Agreed. One way would be to put the camera in the freezer then take darks and monitor camera temp. Capture/stack dark frames in groups as temperature rises every 5 degrees. Hopefully enough frames would be capture in each group for the entire range of temperatures to make a reasonable sized dark frame library. Another way would be to allow master darks to be loaded into a dark library folder manually and perform dark frame library creation with a custom external process. Files names would be based on applicable temperature, like 35.fit, 55.fit, 70.fit, etc. Folders would be divided by gains for day, moon, and night, like /day, /moon, /night. I wouldn’t recommend using JPG format due to compression artifacts. I would recommend using PNG or FIT for dark frame extraction before converting file to PNG or JPG.

Also, a way to select dark frame capture mode in the webGUI would be nice.

[aaronwmorris]

My current theory behind dark frames is time based (as opposed to temperature based). Exposures are taken at 5s intervals at the day, night, and moonmode gains. Temperature is also recorded from the camera, although, I do not currently utilize this data point (not all cameras have a temperature sensor). All darks are stored as the original FITS files from INDI.

If you have captured darks using indi-allsky, you can view your dark frame info in a hidden view: https://hostname/indi-allsky/darks The files are normally stored in /var/www/html/allsky/images/darks

The use of the SQLite database makes this easy as I store all of the metadata in the darkframe table. Dark frame model: https://github.com/aaronwmorris/indi-allsky/blob/master/indi_allsky/flask/models.py#L86 I give the files descriptive names, but the file names do not matter since I query the metadata from the database. SQL is much more efficient than searching file names.

I can investigate adding a manual dark frame upload. I can also look at updating the dark frame matching algorithm to include temperature in the matching.

[HxPI]

Normally, darks are taken to match temperature AND exposure time per given gain. I normally expose for 30 sec at gain 250. Depending on time of year, the temperature can vary. I’m not sure 5 second darks are going to properly calibrate with my given exposure settings. Of course this is so much easier with a cooled camera, which I’m not using. I suppose worse case I can just save raw FITs and make my own dark library. Perhaps an option to utilize a custom dark library folder with matching algorithm would be useful. Also need to verify whether FITs orientation (ROWORDER) is BOTTOM-UP or TOP-DOWN. Anything you can implement to assist with this is greatly appreciated. Thanks.

[aaronwmorris]

Sorry, I was not clear.... The darks are taken in 5 second increments, 1, 5, 10, 15, 20, etc until the max exposure time. The dark is matched at the next highest exposure found. ie a 21 second exposure will have the 25 second dark applied to it.

I have just not found temperature matched darks to be critical for all sky images. I know they are very important with deep sky images, but the signal is so low with deep sky, temperature matched darks are necessary to extract as much signal as possible. All sky images are usually fully exposed and are rarely stretched.

But... I am not here to tell people how to use their cameras. :-) I am looking at how to implement darks with temperature matching.

[HxPI]

I see what you mean now. You are correct all sky images should not be linear like deep sky. My primary concern is amp glow and hot pixels. Your method should captures those artifacts so that should work. I’ll take a look at this when I get my new RPi4 up and running. Thanks.

[aaronwmorris]

I have not really found the issue with my long compression times with PNG. However, if you reduce the compression level for PNG to 5 or 6 the compression times are reasonable. This does not cause a decrease in quality since the compression is lossless, but the files are slightly larger.

I have also completed a new script to setup automounting of USB drives when there is no GUI running. The setup script is at ./misc/setup_usb_automount.sh. The automount uses the same subsystem as Astroberry, so the mount is setup the same exact way. I will eventually get around to writing a wiki page for it.

I am still working on the temperature matched darks...

[HxPI]

I reinstalled with Raspbian 10 on RPi4. A couple issues. First the wifi isn't working. The hotspot is active but I am unable to connect. I wanted Network Manager installed so I could change some settings but not sure if that would affect wifi hotspot. Also, can you add Network Manager to the install script? It's the only way I know to change the wifi password and other network management tasks. So currently unable to test with wifi. Also now wired network is not working, getting DNS errors during setup script and general web browsing. Very random because it was working before when I tried this same setup. Wasn’t able to verify USB access, I was changing the drive storage location and rerun setup script when the wired network issue was noticed.

I recommend adding the option in hotspot setup script to change Wi-Fi password. Also, there may be a conflict introduced after installing network manager. Please take a look at this and/or add network manager install to the script. I understand all these mods are not within the original scope of the build but they are desirable for a portable rig (VNC, Wi-Fi, remote control, USB drive, PNG/TIFF format, etc). Perhaps a separate build for portable configuration so it doesn’t add complexity to those who just want the original fixed configuration?

I may try a complete reinstall with just everything default and no usb or Wi-Fi and see what that does. If that works I’ll try Wi-Fi and usb. Regardless, I’ll need to wait until these issues are worked out for a final working solution.

[HxPI]

Reinstalled everything default, changed a couple config settings, and so far it’s working as it should. I’ll let it run for a while and see what it does for day and night timelapses. Next step will be run usb script and change storage to usb.

[HxPI]

Can the SQM sky brightness – mags/arcsec² be added to right side of jSQM chart? I have no idea what the left side number is but sky brightness is easier to understand. Thanks

http://unihedron.com/projects/darksky/magconv.php