Memory allocation exceeded

[BigBayWx]

I’ve been running for a few months now and getting a lot of server errors with allocated memory issues. My guess as to the cause of this is not being able to run the cleanup scripts. I’ve never been able to get them to work as they always return a wrong source error even though I’m pretty sure I’ve got it right. What I have done instead is every day or two to FTP into the box and manually delete all the old Emwin/Nws and Goes 16 images and that seems to work. Now suddenly for the last several days I’m getting the memory errors and when I check the status using the Raspberry Pi app on the iPad it’s showing only about 15% of the RAM being used and about the same for disk size.

My guess is also that a lot of the file information is cached somewhere as if I pull up something like my local area forecast discussion it shows a lot of them as well as an old warning about a month old still on the local conditions page. Just where is this information cached in the system? If I can find the folder those files are in I can manually delete them even if I can’t get the cleanup scripts to work.

Thanks in advance and once again a great program.

Doug

[mbruski]

Ditto here. Started about 6 weeks ago. I manually deleted all but 2 months worth of NWS, TXT, EMWIN and Image directories and the issue did not resolve.

[JVital2013]

Hello all,

As is stated at the top of the docs here, it is recommended that you only keep 36 hours of EMWIN text data. Any more than 7 days, and things will slow down significantly and memory issues will start; 2 months of text data is certainly too much. This is because GOES simply transmits too many text files - 40,000-60,000 a day - to enumerate more than a few days at a time. This only applies to the text data due to the sheer volume transmitted - it is safe to keep a few months of imagery.

There is no cache anywhere. EMWIN files are read directly. You mention clearing out old images, but cleaning up text is the thing that matters (and it sounds like this is not happening).

The recommended course of action is to make sure the Cleanup-EmwinText script is running daily around noon local time, which will zip up all text files from the last day - keeping them safely out of reach of Vitality GOES.

I can help troubleshooting what's going wrong, but I'll need more details. What's not working about it? What is the output?

[BigBayWx]

Hi Jamie;

Thanks for taking the time to get back with me. Never have been able to get that script to work and I’ve executed the script from several different directories with the same result. It tells me ‘zip’ is not installed and provides the commands to install it but the zip program is installed but the script for some reason isn't seeing it. And it’s interesting that their isn’t a cache. I’ve been manually deleting yesterdays file every day in order to use the program without the memory issues yet it still is showing a warning over a month old but yet it only has the previous day’s file to draw on. Weird….anyway please see attached and let me know what you think.

Doug



On May 26, 2024, at 4:42 PM, Jamie Vital @.***> wrote:

Hello all,

As is stated at the top of the docs here https://github.com/JVital2013/vitality-goes/blob/main/docs/used-emwin-data.md, it is recommended that you only keep 36 hours of EMWIN text data. Any more than 7 days, and things will slow down significantly and memory issues will start; 2 months of text data is certainly too much. This is because GOES simply transmits too many text files - 40,000-60,000 a day - to enumerate more than a few days at a time. This only applies to the text data due to the sheer volume transmitted - it is safe to keep a few months of imagery.

There is no cache anywhere. EMWIN files are read directly. You mention clearing out old images, but cleaning up text is the thing that matters (and it sounds like this is not happening).

The recommended course of action is to make sure the Cleanup-EmwinText script is running daily around noon local time, which will zip up all text files from the last day - keeping them safely out of reach of Vitality GOES.

I can help troubleshooting what's going wrong, but I'll need more details. What's not working about it? What is the output?

— Reply to this email directly, view it on GitHub https://github.com/JVital2013/vitality-goes/issues/65#issuecomment-2132390030, or unsubscribe https://github.com/notifications/unsubscribe-auth/BGWZT7PHS7ILEPRZYYO5M7DZEJCKNAVCNFSM6AAAAABHRBILGSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMZSGM4TAMBTGA. You are receiving this because you authored the thread.

[JVital2013]

Unfortunately the log did not come though (GitHub often loses attachments when you try to reply via email), but I have some ideas based on what you said:

It tells me ‘zip’ is not installed and provides the commands to install it but the zip program is installed but the script for some reason isn't seeing it.

Ah, I think I know what's going on here - are you running it via cron? If so, I recently found that some systems won't find all commands when running via cron the way I used to recommend. I recently updated the recommended cron file, like this:

0 0 * * * youruser bash -lc /path/to/vitality-goes/scripts/CreateVideos-ABI.sh
0 2 * * * youruser bash -lc /path/to/vitality-goes/scripts/CreateVideos-EMWIN.sh
55 11 * * * youruser bash -lc /path/to/vitality-goes/scripts/Cleanup-EmwinText.sh
25,55 2-23 * * * youruser bash -lc /path/to/vitality-goes/scripts/Sanchez.sh

The bash -lc is required to get the correct environment variables, which was missing from my previous instructions.

I’ve been manually deleting yesterdays file every day in order to use the program without the memory issues yet it still is showing a warning over a month old but yet it only has the previous day’s file to draw on.

Then not everything is getting deleted. Is your EMWIN data being saved to dated subfolders, like goestools lets you do? If so, a bunch of the files end up in a folder dated 1970-01-01 or 1969-12-31. This is a goestools bug. Delete that old folder and see if the Vitality GOES error remains.

[JVital2013]

Quick follow-up: I'd recommend focusing on finding where the stray emwin files are before getting the script going. The cleanup script does not work retroactively at this time.

[BigBayWx]

Hi Jamie;

Posting this on github now and just including the screenshots that didn't go through before. The stray files were posting in the 1969-12-31 folder as you had surmised. I've never been totally able to delete that file as of course goestools keeps posting to it. Took me awhile to figure out how to empty it totally and now the app is working snappy and if I can't get the script going I'll just delete it manually every day and it should work fine.

As far as the script goes I haven't set up the cron yet as I was trying to call the script manually first. I also have a copy of the scriptconfig.ini in /var/www/html/config with the rest of the config files. I put a copy of it in the home folder just to play around and see if it made a difference in executing the script which it did not.

[JVital2013]

It looks like you're calling the scripts with sh - the scripts are bash scripts as is designated in their headers, and will not run correctly with sh.

Try calling them without specifying sh, like ./Cleanup-EmwinText.sh or bash Cleanup-EmwinText.sh

[BigBayWx]

Jamie;That was it.  Running it with using bash instead of sh runs the script and leaves a 40mb file of yesterdays Emwin.  It doesn’t delete the daily folder or do anything with the 1969-12-31 folder though.  If that is the way its supposed to work then it is working as expected and I thank you for your help.DougSent from my iPadSent from my iPadOn May 27, 2024, at 11:01, Jamie Vital @.***> wrote: It looks like you're calling the scripts with sh - the scripts are bash scripts as is designated in their headers, and will not run correctly with sh. Try calling them without specifying sh, like ./Cleanup-EmwinText.sh or bash Cleanup-EmwinText.sh

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: @.***>

[JVital2013]

Yes, it does not delete the folders themselves due to the unreliability in which goestools names them - it only deletes the files themselves. I've found it is better to just ignore the folders, or disable them entirely. If you remove the date template from your emwin handler in goesproc-goesr.conf file, like this...

# EMWIN
[[handler]]
type = "emwin"
directory = "/path/to/goestoolsdata/emwin"

...it will save data directly in the EMWIN folder. Doing so would also let you set fastEmwin to true in your Vitality GOES config.ini, which speeds things up.