owntracks / recorder

Store and access data published by OwnTracks apps
Other
894 stars 123 forks source link

Raspberry Pi 2: can't launch ot-recorder ("Cannot allocate memory") #480

Closed neurolit closed 3 months ago

neurolit commented 3 months ago

Hi!

I just installed OwnTracks on my Raspberry Pi 2 (with freshly installed Raspberry Pi OS) using quick setup and bootstrap.sh. Everything went well, with no error.

However, part of OwnTracks does'nt work: ot-recorder service isn't started, with this error message:

Jul 26 07:35:23 raspberrypi systemd[1]: Starting ot-recorder.service - OwnTracks Recorder...
Jul 26 07:35:26 raspberrypi systemd[1]: Started ot-recorder.service - OwnTracks Recorder.
Jul 26 07:35:26 raspberrypi ot-recorder[32074]: version 0.9.8 starting with STORAGEDIR=/var/spool/owntracks/recorder/store
Jul 26 07:35:26 raspberrypi ot-recorder[32074]: gcache_open: mdb_env_open: Cannot allocate memory
Jul 26 07:35:26 raspberrypi ot-recorder[32074]: Can't initialize gcache in /var/spool/owntracks/recorder/store/ghash
Jul 26 07:35:26 raspberrypi systemd[1]: ot-recorder.service: Main process exited, code=exited, status=1/FAILURE
Jul 26 07:35:26 raspberrypi systemd[1]: ot-recorder.service: Failed with result 'exit-code'.

I thought using the solution described in #404 and #348 would solve this problem, but it doesn't. I add Environment="OTR_LMDBSIZE=10485760" in [Service] part of /etc/systemd/system/ot-recorder.service, deleted /var/spool/owntracks/recorder/store/ghash/*.mdb files, exported OTR_LMDBSIZE=10485760 in the shell before running ot-recorder --initialize (which created new mdb files), but when I start ot-recorder (with sudo systemctl start ot-recorder), the same error happens.

Do you know what I could try in order to run ot-recorder?

jpmens commented 3 months ago

I'm sorry you're having trouble getting that to start!

I've not seen an Rpi 2 for a bit, so I'm just guessing. Can you please try lowering that cache size a bit, say

rm -f /var/spool/owntracks/recorder/store/ghash/*.mdb
export OTR_LMDBSIZE=1048576  # 1MB
ot-recorder --initialize

ot-recorder   # launch it on the console (for testing, no need to use service)

and see whether it will then start?

neurolit commented 3 months ago

Thanks @jpmens for your help.

It started perfectly:

ot-recorder[10170]: version 0.9.8 starting with STORAGEDIR=/var/spool/owntracks/recorder/store
ot-recorder[10170]: connecting to MQTT on 127.0.0.1:1883 as clientID otrec without TLS
ot-recorder[10170]: HTTP listener started on 127.0.0.1:8083, without browser-apikey
ot-recorder[10170]: HTTP prefix is https://[redacted]/owntracks
ot-recorder[10170]: Using storage at /var/spool/owntracks/recorder/store with precision 7
ot-recorder[10170]: TZDATADB is at /usr/share/owntracks/recorder/timezone16.bin: R_OK
ot-recorder[10170]: Subscribing to owntracks/# (qos=2)

Should I try to change environment variable in /etc/systemd/system/ot-recorder.service and try to start the service?

jpmens commented 3 months ago

Yes, please do, and I don't doubt (famous last words) it will work.

Depending on how many reverse-geo entries you will produce over time (I am assuming you configured OpenCage with a key), the space we've reserved for LMDB might become a bit tight, but we'll have to cross that bridge when we get there.

You might like to try 2MB

export OTR_LMDBSIZE=2097152
...
neurolit commented 3 months ago

I tried 2097152, and it worked when I recreated the mdb files, and when I launched (without the service) ot-recorder. Good.

However, after having updated /etc/systemd/system/ot-recorder.service, launched sudo systemctl daemon-reload and sudo systemctl start ot-recorder, the same problem arises:

Jul 26 14:17:50 raspberrypi systemd[1]: Starting ot-recorder.service - OwnTracks Recorder...
Jul 26 14:17:53 raspberrypi systemd[1]: Started ot-recorder.service - OwnTracks Recorder.
Jul 26 14:17:53 raspberrypi ot-recorder[10653]: version 0.9.8 starting with STORAGEDIR=/var/spool/owntracks/recorder/store
Jul 26 14:17:53 raspberrypi ot-recorder[10653]: gcache_open: mdb_env_open: Cannot allocate memory
Jul 26 14:17:53 raspberrypi ot-recorder[10653]: Can't initialize gcache in /var/spool/owntracks/recorder/store/ghash
Jul 26 14:17:53 raspberrypi systemd[1]: ot-recorder.service: Main process exited, code=exited, status=1/FAILURE
Jul 26 14:17:53 raspberrypi systemd[1]: ot-recorder.service: Failed with result 'exit-code'.

I tried with 1048576, same.

:sob:

So maybe the environment value isn't used by service? Here is my service file:

# /etc/systemd/system/ot-recorder.service

[Unit]
Description=OwnTracks Recorder
Wants=network-online.target
After=network-online.target

[Service]
Environment="OTR_LMDBSIZE=1048576"
Type=simple
User=owntracks
WorkingDirectory=/
ExecStartPre=/bin/sleep 3
ExecStart=/usr/sbin/ot-recorder

[Install]
WantedBy=multi-user.target
jpmens commented 3 months ago

Did you run

systemctl daemon-reload

after changing the unit file? Yes, you did.

Do you see your change with

systemctl cat ot-recorder

?

jpmens commented 3 months ago

Oopsie!

        if ((p = getenv("OTR_LMDBSIZE")) != NULL) {
                lmdb_size = atol(p);
                if (lmdb_size < 10485760) { // 10 MB
                        lmdb_size = 10485760;
                }
        }

If the configured size is below 10MB we set it to 10MB, hence what we're doing here has no effect.

Are you able to compile code on your Rpi? I could provide a patch quickly.

neurolit commented 3 months ago

Oh oh: systemctl cat ot-recorder gives:

# /etc/systemd/system/ot-recorder.service

[Unit]
Description=OwnTracks Recorder
Wants=network-online.target
After=network-online.target

[Service]
Environment="OTR_LMDBSIZE=1048576"
Type=simple
User=owntracks
WorkingDirectory=/
ExecStartPre=/bin/sleep 3
ExecStart=/usr/sbin/ot-recorder

[Install]
WantedBy=multi-user.target

# /etc/systemd/system/ot-recorder.service.d/override.conf
[Service]
# works as environment variable only (not in ot-recorder defaults)
Environment="OTR_LMDBSIZE=5368709120"

Where does this override.conf comes from?

jpmens commented 3 months ago

It's created during Quicksetup config, but that's not the problem, sadly. See above for my previous "Oopsie" comment.

neurolit commented 3 months ago

If the configured size is below 10MB we set it to 10MB, hence what we're doing here has no effect.

Are you able to compile code on your Rpi? I could provide a patch quickly.

I tried manually with 10MB, and it works! So the problem comes from this override.conf, and I could go with 10MB (without a patch, then).

The value set by override.conf is too large, but 10MB is OK.

jpmens commented 3 months ago

Thanks for finding that! (You know my code better than I do -- tbh I'd forgotten about the override.) I have submitted a patch now which will in future print the desired size of the lmdb cache; we'll be able to pinpoint the source more easily, so thanks for that!

Very glad you got it sorted.

jpmens commented 3 months ago

Keep in mind, please, that when you re-run ./bootstrap.sh in Quicksetup, your 10MB will be overwritten. I will apply a fix to Quicksetup in order to avoid doing so if the file exists.

neurolit commented 3 months ago

Thank you very much for your help! I updated /etc/systemd/system/ot-recorder.service.d/override.conf (by using 10MB) and it works now :champagne:

jpmens commented 3 months ago

Thank you for your help. I think we've added two (very simple but useful) changes.

rabbit-11177

If you'll please run a git pull in your Quicksetup directory, you'll be set.