search

filipsPL / autowx2

The program for scheduling recordings and processing of the satellite and ground radio transmissions (like capturing of the weather APT images from NOAA satellites, voice messages from ISS, fixed time recordings of WeatherFaxes etc.) :earth_africa: :satellite:
78 stars 16 forks source link

Various improvements and bug-fixes #91

Open GlassesToCSharp opened 4 years ago

GlassesToCSharp commented 4 years ago

This PR contains a variety of improvements and bug-fixes that were found when running on a Raspberry Pi 3 Model B (B+?).

Major Changes

Replaced Flask server with NodeJS

This change follows after the RPi was unable to run sox with an instance of the Flask webserver. It was only after removing the thread running the flask server that recording and processing worked on the RPi. This was raised here.

The alternative was to use a NodeJS server. Why use NodeJS instead of fixing the Flask instance?

Using NodeJS, I now have two terminals: one running the web server instance, and the other running AutoWX2. Interupting one will not affect the other.

To set up a NodeJS server on a Raspberry Pi, there is a very good tutorial by thisDaveJ that takes through all the steps to setting it up.

subprocess.Popen()

According to this SO answer, using subprocess.Popen() creates a new process with the same memory footprint as the Python code calling it. I noticed after a couple of days (sometimes a few hours!) of running AutoWX2 continuously that I was getting OSError: [Errno 12] Cannot allocate memory when running any bash script. That same SO answer recommends creating a single instance of subprocess.Popen() at the beginning of your Python script, and calling it with the appropriate command.

I created a new file called shell_scripts.sh which will run the command you give it until it completes. This bash script is called at the beginning of autowx2_function.py only once, so that only a small memory footprint is used. I have not seen memory issues since implementing this.

Note: this is probably screaming "memory leaks" to the average developer. I have not investigated this, so the underlying memory leakage problem might still be there.

Other changes

I am accessing my RPi's terminal through SSH, and I have byobu installed to access multiple terminals through one SSH channel. I also have RPi-Monitor running to check memory consumption, CPU usage/temperature, and other information as the RPi runs.

I have not tested this PR on desktop machines, so I have no idea what will happen on them. I can only hope good things!

I'm not a backend/Python/Bash/Linux/JavaScript guru, so some of the code may look messy and might not conform to "standards". All the changes I made were to improve the performance on my Raspberry Pi and fix bugs that I had noticed.

GlassesToCSharp commented 4 years ago

@filipsPL I give up. I don't have sufficient knowledge to understand how to fix the last Codacy issue. Can I ask for your help on this please?

filipsPL commented 4 years ago

@ThomasBartleet thanks for the great effort you put into developement of autowx2! :+1: Good job! I didn't have time to look at it before, but seems to lookk very good. My only doubt is if these changes will fit only Raspberry Pi or the code will work on any system? This must be checked before the final merge.

About codacy - this one (popen) is indeed a pain in the ... neck. If this is the only issue, I guess we may have it there.

Maybe we should consider creating another pi-oriented branch for autowx2, and after we make sure the version works fine for all systems, merge it with the main branch?

GlassesToCSharp commented 4 years ago

@filipsPL I think the changes should benefit any system, not just the RPi, but I do agree that this PR should be tested on other systems. I did try to keep it all as generic as possible! The only bigger change is the implementation of the Node.js server, but there seem to be tutorials on how to set up Node.js on multiple systems (Windows, Mac, Ubuntu, Beaglebone etc), so I don't think that should be a problem.

On another note, is there a setting in Codacy that you can tell it to ignore the Popen() warning/error?

Regarding creating a RPi-orientated branch, I think the only only differences between using AutoWX2 on an RPi and any other system is the installation. Otherwise, everything else is pretty much the same. Up to you! :)

filipsPL commented 4 years ago

@ThomasBartleet thanks! I will check the code then and try to run it on my ubuntu. Also look at the codacy to ignore Popen().

Maybe other could also check this version on their system?