Publishing works, but Recorder UI does not show anything

[sidamos]

I have installed owntracks recorder in Docker with a Caddy reverse proxy in front. I am using HTTPS to publish events. I have only configured an MQTT broker, so that the recorder would start up.

In the iOS app, I have configured https://owntracks.mydomain.org/pub and enabled basic auth.

According to the app logs, publishing worked with HTTP 200 status.

When I look into the /store volume, I can see: store/rec/thomas/3930c0cf-dfa8-4d7c-86a5-ee5910bd443f/2024-07.rec

Inside, there is: 2024-07-28T08:02:16Z * {some JSON wich looks good}

My problem is, the web UI of the recorder, shows nothing.

In the recorder docker container log, I see:

• http: GET /
• http: GET /api/0/last /store/last: No such file or directory

Any hints would be greatly appreciated.

[jpmens]

I assume there's something wrong with the reverse proxy configuration, as you say that you can publish from iOS to the Recorder.

Maybe the nginx reverse proxy configuration we demonstrate in this project's README.md can shed some light?

BTW, you do not need to run an MQTT broker if you don't want it: launching ot-recorder with option --port 0 disables the requirement to connect to an MQTT broker.

[sidamos]

Thanks for the quick response!

The iOS device publishes over the reverse proxy, so I assume that it is working OK. I am using the Caddy reverse proxy config from your docs.

I was not clear about my problem. The web UI of the recorder does work, but it shows no users/devices, although I did publish from the iOS app and I can see the published data in /store. This issue is the same for using the web UI over the reverse proxy or locally.

So, the question is: Why does the recorder web UI not show the data that I can see in /store?

[jpmens]

/store/last: No such file or directory

worries me a bit. Does that file really not exist? Also not after you manually publish a location from iOS?

[sidamos]

I have only this in /store: ghash monitor rec

[jpmens]

Please stop the Recorder and run ot-recorder --initialize (this is non-destructive), and then publish a location again. Does the ../last/ directory show up?

[sidamos]

How do I do that? I am running it in Docker.

[sidamos]

I could do it by enter the image, but for that, the image must be running.

[sidamos]

OK, so I entered the container and did "ot-recorder --initialize" and then restarted the container and published again. Still no "/store/last". I have now multiple entries in "store/rec/thomas/3930c0cf-dfa8-4d7c-86a5-ee5910bd443f/2024-07.rec", so publishing seems to be working fine.

[jpmens]

Do you see any errors in the Recorder's log (recorder docker container log) since the Recorder started? If so, which?

[sidamos]

No errors. I now manually created store/last, so it does not even complain about that anymore. Still I see no user location data in the web UI after publish.

+ version 0.9.8 starting with STORAGEDIR=/store
+ Not using MQTT: disabled by port=0
+ HTTP listener started on 0.0.0.0:8083, without browser-apikey
+ HTTP prefix is unset
+ Using storage at /store with precision 7
+ TZDATADB is at /config/timezone16.bin: ENOENT
+ http: POST /pub
+ http: GET /
+ http: GET /api/0/last

Hm, it seems to have a time zone issue.

[sidamos]

Maybe, I did something silly during setup, so I am providing my full config (I have added the time zone stuff to the docker compose file that I found on the internet because of the ENOENT error, but it did not solve that).

docker-compose.yml:

version: '3'

services:
  otrecorder:
    image: owntracks/recorder
    ports:
      - 8083:8083
    volumes:
      - $PWD/config:/config
      - $PWD/store:/store
      - "/etc/timezone:/etc/timezone:ro"
      - "/etc/localtime:/etc/localtime:ro"
    environment:
      - TZ=Europe/Berlin
    restart: unless-stopped

networks:
  default:
    name: owntracks-network

config/recorder.conf:

OTR_HTTPHOST = "0.0.0.0"
OTR_PORT = 0

[jpmens]

What you see in below screenshot is the result of what roughly corresponds to what you are attemtping, albeit without the reverse proxy.

docker run -p 8083:8083 -v /tmp/sidamos:/store -e OTR_PORT=0 owntracks/recorder

1. The error about missing /store/last shows (I ought to fix that), but the directory is created automatically
2. I POST a location to /pub
3. I then launch a Web browser on 127.0.0.1:8083 to get the "UI", which appears

Maybe try those steps first and then attempt to configure your reverse proxy.

[sidamos]

Good idea. I did exactly what you wrote and configured the iOS app to use http://192.168.0.7:8083/pub (Docker runs on that host). This is what I get:

# docker run -p 8083:8083 -v /tmp/sidamos:/store -e OTR_PORT=0 owntracks/recorder
+ version 0.9.8 starting with STORAGEDIR=/store
+ Not using MQTT: disabled by port=0
+ HTTP listener started on 0.0.0.0:8083, without browser-apikey
+ HTTP prefix is unset
+ Using storage at /store with precision 7
+ TZDATADB is at /config/timezone16.bin: R_OK
+ http: POST /pub
+ http: GET /
+ http: GET /api/0/last
/store/last: No such file or directory

So, now it is happy about the time zone, but:

# ls /tmp/sidamos/
ghash  monitor  rec

The "last" directory was NOT created and the web UI still shows no devices/users.

[jpmens]

So, please start your last test again, but instead of using your iOS device to publish, let's use this little script which uses a pseudo location in Paris. That way you won't divulge any data :-)

#/bin/sh

tics=$(date +%s)

# paris
lat=48.85833
lon=3.29513

payload="{\"_type\":\"location\",\"lat\":$lat,\"lon\":$lon,\"t\":\"u\",\"tid\":\"SC\",\"tst\":$tics}"

echo $payload

curl -sSf --data "${payload}" "http://192.168.0.7:8083/pub?u=sidamos&d=curl"

I'd then like to see the whole stderr log please, from start of the docker image to when you've performed the HTTP POST with the above.

In addition, I'd then like to see the result of

$ curl http://192.168.0.7:8083/api/0/version

$ curl http://192.168.0.7:8083/api/0/last

[sidamos]

Looks like it is the iOS app's problem. It now really created the "last" directory.

# docker run -p 8083:8083 -v /tmp/sidamos:/store -e OTR_PORT=0 owntracks/recorder 2>&1
+ version 0.9.8 starting with STORAGEDIR=/store
+ Not using MQTT: disabled by port=0
+ HTTP listener started on 0.0.0.0:8083, without browser-apikey
+ HTTP prefix is unset
+ Using storage at /store with precision 7
+ TZDATADB is at /config/timezone16.bin: R_OK
+ http: POST /pub
/store/last: No such file or directory
- 11:07:18 owntracks/sidamos/curl              t=u tid=SC loc=48.85833,3.29513 [__] Unknown (48.858330,3.295130) (u0dmfyr)

# curl http://192.168.0.7:8083/api/0/version
{"version":"0.9.8","git":"0.9.8-0-g488eabe3e3"}

# curl http://192.168.0.7:8083/api/0/last
[{"_type":"location","lat":48.85833,"lon":3.29513,"t":"u","tid":"SC","tst":1722164838,"_http":true,"topic":"owntracks/sidamos/curl","username":"sidamos","device":"curl","ghash":"u0dmfyr","isotst":"2024-07-28T11:07:18Z","disptst":"2024-07-28 11:07:18","tzname":"Europe/Paris","isolocal":"2024-07-28T13:07:18+0200"}]

And the UI shows one line for the user. Now the question is: What is the problem of the iOS, so that I can fix it?

[jpmens]

Judging by what I saw above, you've not configured a devicename in iOS, only a username (thomas). Could you please set a device name in the OwnTracks iOS settings, such as myphone and try again?

[sidamos]

There is no "devicename", only "DeviceID" and that was set by the app automatically to an UUID.

[jpmens]

Yes, device ID. Set that to a value you chose. (meinIphone, etc.)

[jpmens]

Alternatively change the URL to a value with u= and d= (username, device), such as what we used in the shell script, e.g.

http://192.168.0.7:8083/pub?u=thomas&d=myphone

[sidamos]

I deleted the container and the store dir and set the DeviceId to iPhoneThomas and tried again. This does not fix the issue. Again, I have no "last" directory and the UI shows no devices. The only difference to before is that I now have "/tmp/sidamos/rec/thomas/iphonethomas" instead of the UUID.

[sidamos]

I deleted the DeviceId again in the iOS app settings and added "?u=thomas&d=iPhone" to the URL. This did not result in a device directory "rec/thomas/iPhone". Instead, it used the UUID again.

[jpmens]

Please set DeviceID again and export your iOS OwnTracks configuration. I assume it has no secrets in it in which case you can please paste i t here or alternatively send it to us at support@owntracks.org

[sidamos]

I have set the password to "abc" and then disabled authentication (which I only need for the reverse proxy anyway) and then tried again and still the same thing, so here is the config dump:

# cat /tmp/sidamos/config/thomas/iphone/thomas-iphone.otrc 
{
    "username": "thomas",
    "maxHistory": 0,
    "locked": false,
    "_type": "configuration",
    "monitoring": 1,
    "deviceId": "iPhone",
    "positions": 50,
    "ranging": false,
    "cmd": false,
    "encryptionKey": "",
    "allowRemoteLocation": true,
    "pubTopicBase": "",
    "tid": "",
    "url": "http://192.168.0.7:8083/pub",
    "ignoreStaleLocations": 0,
    "osmCopyright": "",
    "waypoints": [],
    "usePassword": false,
    "httpHeaders": "",
    "auth": false,
    "locatorInterval": 180,
    "extendedData": true,
    "osmTemplate": "",
    "ignoreInaccurateLocations": 0,
    "locatorDisplacement": 200,
    "mode": 3,
    "password": "abc",
    "downgrade": 0
}

[jpmens]

Please configure TID. That's the tracker ID, a 2-character combination whcih later shows up on the map (unless you have a CARD configured). I think that might be the issue...

TID might be your initials, say.

[sidamos]

This did not help. The iOS app had set that to a random string ("NE") anyway. I overwrote that with "TB" and tried from scratch and same issue.

[jpmens]

(yes, "NE" are the last two letters of the DeviceID)

[jpmens]

Let's try some debugging....

Please stop the container and launch in foreground as follows (stop it with CTL-C if needed):

docker run -it --entrypoint /usr/sbin/ot-recorder -p 8083:8083 -v /tmp/sidamos:/store -e OTR_PORT=0 owntracks/recorder --debug

Then please publish a location from your iPhone and paste the output here (or mail it to support@onwtracks.org if you don't wish to edit out lat / lon for privacy.

[sidamos]

iPhone is my main phone, but out of curiosity, I just tested the the app on Android and there it works fine. Will now test what you just wrote.

[jpmens]

To be honest, I can't recall when our iOS app caused so much work -- I use it permanently. (In MQTT mode :-) )

[sidamos]

I emailed the output.

[jpmens]

Thank you, @sidamos for your great and timely help in debugging this.

--- REQUEST --------------------------- /pub (587)
{"_type":"status","iOS":{"deviceSystemName":"iOS", ... [redacted]

What's happening is that iOS is sending us one message only, and it's not a location message (which should be "_type" : "location") but is a new "_type" : "status" message which was introduced in March of this year.

Simultaneously, we've uncovered a bug in the Recorder, which doesn't explicitly handle _type: status, though that isn't what's causing the problem.

I'm going to hand off to @ckrey to ask why iOS is begining with a _type: status instead of a _type: location.

[sidamos]

Thanks! I guess, switching to MQTT would help, since you mentioned it does work for you.

But then I need to figure out how to expose my MQTT securely to the internet with auth and SSL (letsencrypt), given that I already run Caddy which occupies the HTTP ports for ACME for letsencrypt.

[jpmens]

If Caddy exposes the Let's Encrypt key, cert, cacert files, you can re-use those (file system permissions allowing) with which to configure the Mosquitto broker, say. You'd also need to open an additional port (8883 is default for MQTT/TLS, but that's configurable.

Or you wait until we sort this out for you, which could, admittedly, take a bit. :-)

[sidamos]

I could re-use another cert, that is correct. I would however also have to enable auth for my MQTT broker and then reconfigure all existing local network clients to do auth. I'll think about it.

[sidamos]

BTW, can you tell me why the recorder hat that error with the time zone only when running via docker compose, but not when running with standard docker command?

[jpmens]

You're asking the one person in the whole wide Internet who knows next to nothing about docker ...

Our image comes bundled with a file for performing timezone lookups in order to populate tzname in the geo cache so that we know in which $tz a location was published. (Can be used to find out later "oh yeah I remember, that was when I was in Canada, so location 12:00 is Canadian time, it must have been lunch" or something.)

The docker compose file you used doesn't configure the path to that correctly.

Please please please don't say it's our fault? ;-) OK, do say so, but I'll have to think on how to fix it.

[jpmens]

volumes:
      - $PWD/config:/config

That overrides the whole of /config/*, doesn't it? That's likely the issue, as the binary file is in /config/timezone16.bin in the image.

[sidamos]

Thanks, I'll check that later.

BTW, I have now tried with MQTT instead of HTTP and have the same issue with the iOS client. I am seeing new entries in the .rec file for each publish, but it does not create the "last" directory.

[jpmens]

Does the UI show the data? That was the original issue. It might also be that the last/ directory is created when the API actually tests for it. (I can't look just now.)

[sidamos]

It is exactly the same issue with MQTT as it was with HTTP. Same effect/behavior. UI does not create the last directory. I have tested with your first command for Docker in the foreground.

Which makes me wonder why MQTT is working for you with the iOS app. Maybe it is a configuration problem in the iOS app after all.

[ckrey]

I found the problem: It is probably not docker or the recorder or the configuration of the iOS app or the iOS app itself. By pressing the publish button on the status screen you are publishing a status message, not a location messages.

To manually publish a location message, press the top right action button on the map screen as shown in the screenshots below

[sidamos]

LOL, it works. ;-) Tested with HTTP.

[sidamos]

BTW, I solved the missing timezone file by once using the suggested Docker config and then copying the file out and into my local config dir. Then switched back to my Docker config. Maybe owntracks wanted to copy the file to the config dir but could not because of permissions since owntracks is not running as root.

[jpmens]

Thank you, @ckrey: this was causing me sleeplessness, and I couldn't for the life of me figure out why the status messages were being sent!

@sidamos glad you're all sorted out.

[sidamos]

Thanks a lot for helping!