search

owntracks / docker-recorder

Docker image for OwnTracks Recorder
151 stars 68 forks source link

updates from Android are happening but no data is visible #76

Closed MrColumbo closed 8 months ago

MrColumbo commented 8 months ago

On the android client it looks all fine ....these are the last lines from the log (i get like a 200OK when a location is transmitted)

2024-03-05 10:44:58.484 I ServiceStarter$Impl: starting service 2024-03-05 10:44:58.492 D MapFragmentFactory: Instantiating Fragment for org.owntracks.android.ui.map.MapFragment 2024-03-05 10:44:58.494 D MapViewModel: setting view mode: VIEW_FREE 2024-03-05 10:44:58.516 D GoogleMapFragment: Maps SDK initialized with renderer: LEGACY 2024-03-05 10:44:58.554 D MapActivity$serviceConnection: Service connected to MapActivity 2024-03-05 10:44:58.554 D LocationLiveData: Removing previous locationupdate task complete. Success=true Cancelled=false 2024-03-05 10:44:58.555 D GoogleMapFragment: Maps SDK initialized with renderer: LEGACY 2024-03-05 10:44:58.555 D GoogleMapFragment: GoogleMapFragment initMap hasLocationCapability=true 2024-03-05 10:44:58.557 D MapViewModel: setting view mode: VIEW_FREE 2024-03-05 10:44:58.557 D GoogleMapFragment: Drawing regions on map 2024-03-05 10:44:58.559 D GoogleMapFragment: Maps SDK initialized with renderer: LEGACY 2024-03-05 10:44:58.559 D GoogleMapFragment: GoogleMapFragment initMap hasLocationCapability=true 2024-03-05 10:44:58.560 D GoogleMapFragment: Drawing regions on map 2024-03-05 10:44:58.567 D LocationLiveData: LocationLiveData location update request completed: Success=true Cancelled=false 2024-03-05 10:44:58.599 D LocationLiveData: Removing previous locationupdate task complete. Success=true Cancelled=false 2024-03-05 10:44:58.626 D LocationLiveData: LocationLiveData location update request completed: Success=true Cancelled=false 2024-03-05 10:45:00.765 D LocationProcessor: ignoring initial or duplicate transition: zu Hause 2024-03-05 10:45:00.772 D MessageProcessor: Queueing messageId:1709631900768-186244, queueLength:0, ThreadID: Thread[main,5,main] 2024-03-05 10:45:00.776 D MessageProcessorEndpointHttp: url:https://##########.duckdns.org/, messageId:1709631900768-186244 2024-03-05 10:45:00.778 D GoogleMapFragment: Drawing regions on map 2024-03-05 10:45:00.778 D CachingGeocoder: Geocode cache: hits=4, misses=10 2024-03-05 10:45:00.845 I MessageProcessorEndpointHttp: Message id=1709631900768-186244 sent in 68ms 2024-03-05 10:45:00.847 E MessageProcessorEndpointHttp: JsonParseException HTTP status: 200 2024-03-05 10:45:00.848 D MessageProcessor: message:HTTP status 200, JsonParseException, 2024-03-05 10:45:00.848 D MessageProcessor: onMessageDelivered in MessageProcessor Noop. ThreadID: Thread[networkHandlerThread,5,main] 2024-03-05 10:45:03.991 I ContactsViewModel: Refreshing contacts geocodes 2024-03-05 10:45:04.249 D LocationLiveData: LocationLiveData removing location updates completed: Success=true Cancelled=false

But when I access the webinterface - via 8083 or the frontend which i installed as well i do not see any data

Even all the logs from recorder look fine - at least I did not spot any error

What am I missing?

jpmens commented 8 months ago

You don't give us much to go on. App versions, Recorder version ,type of connection (I infer HTTP), how installed, ...

Verify whether Recorder is getting data by looking if the file /var/spool/owntracks/recorder/rec/<username>/<devicename>/YYYYMM.rec is getting data. Do you see locations in that file?

MrColumbo commented 8 months ago

there is nothing in /var/spool ...no owntracks directory

I am using the latest version via docker

version: "3" services: recorder: ports:

There is a reverse proxy in front using basic authentication and pointing to port 8083 on the server

jpmens commented 8 months ago

I don't see you've configured an incoming port for OwnTracks over HTTP. Your log above shows you connect to 443, but how is that port coupled with your Docker container?

MessageProcessorEndpointHttp: JsonParseException

indicates the app is getting something back, but not what it's expecting.

Look at the examples and keep an eye on port 8083; that's where your app will (eventually) need to speak to, but note that's not an HTTPS (SSL) endpoint but a simple HTTP endpoint.

I know this sounds very complicated, and it is. As such I would very much like to recommend you study what our new Quicksetup can do and the features it offers. Basically everything out of the box, including configuration of your Android (or iOS) apps.

MrColumbo commented 8 months ago

In the reverse proxy i am connecting https 443 to http 8083

in the reverse proxy i see

05/Mar/2024:10:35:50 +0000] - 200 200 - POST https ##########.duckdns.org "/" [Client 109.250.xxx.yyy] [Length 1272] [Gzip 2.55] [Sent-to 192.168.178.188] "Owntracks-Android/420412000" "-" [05/Mar/2024:11:58:46 +0000] - 200 200 - POST http ##########.duckdns.org "/" [Client 109.250.xxx.yyy] [Length 1272] [Gzip 2.55] [Sent-to 192.168.178.188] "Owntracks-Android/420412000" "-" [05/Mar/2024:11:58:46 +0000] - 200 200 - POST http ##########.duckdns.org "/" [Client 109.250.xxx.yyy] [Length 1272] [Gzip 2.55] [Sent-to 192.168.178.188] "Owntracks-Android/420412000" "-" [05/Mar/2024:11:58:46 +0000] - 200 200 - POST http ##########.duckdns.org "/" [Client 109.250.126.239] [Length 1272] [Gzip 2.55] [Sent-to 192.168.178.188] "Owntracks-Android/420412000" "-"

Is the client supposed to access the same web port as i use for the basic UI?

When I use the browser and connect to https://...... i see the page with that stuff Live MAP of current locations

User/Device Points Points Track Track Track

....is the client when using http is supposed to send data to this URL?

(just a sidenote ..since i am using proxmos i just spun up a debian 12 lxc and tried the quicksetup but it failed to start mqqt .....i will open a seperate issue for that).

By the way ...i noticed in the quicksetup there is a file where i put in the names of the users in a yaml file ......not sure how they will be used since somehow i understood that i need to use basic authentication on the reverse proxy and that way the user will get passed to the recorder. But maybe the yaml file is about taking care of that portion.

jpmens commented 8 months ago

Is the client supposed to access the same web port as i use for the basic UI?

No, the Android app will access /pub

We have an example nginx proxy configuration at https://github.com/owntracks/recorder?tab=readme-ov-file#nginx which might help you.

When I use the browser and connect to https://...... i see the page with that stuff

That sounds right.

(just a sidenote ..since i am using proxmos i just spun up a debian 12 lxc and tried the quicksetup but it failed to start mqqt .....i will open a seperate issue for that).

Please do: the issue tracker for that is at https://github.com/owntracks/quicksetup/issues but I'm sure you've found it. :-)

By the way ...i noticed in the quicksetup there is a file where i put in the names of the users in a yaml file ......not sure how they will be used since somehow i understood that i need to use basic authentication on the reverse proxy and that way the user will get passed to the recorder. But maybe the yaml file is about taking care of that portion.

The users (we call them 'friends') indeed perform basic auth with their user/password against nginx. From then on they can see (e.g. in Frontend) all users' data. Likewise these friends use the name username and passwords for the MQTT connections.

MrColumbo commented 8 months ago

Since it looks like i am not that far away from getting this runnig i did a tshark to see the request and respoinse...

Request looks like that

 POST / HTTP/1.1\r\n
        Request Method: POST
        Request URI: /
        Request Version: HTTP/1.1
    Connection: Keep-Alive\r\n
    Host: #########.duckdns.org\r\n
    X-Forwarded-Scheme: http\r\n
    X-Forwarded-Proto: http\r\n
    X-Forwarded-For: 109.250.xxx.yyy\r\n
    X-Real-IP: 109.250.xxx.yyy\r\n
    Content-Length: 307\r\n
        [Content length: 307]
    User-Agent: Owntracks-Android/420412000\r\n
    X-Limit-U: hotzenplotz\r\n
    X-Limit-D: hotzenplotz\r\n
    Cache-Control: no-cache\r\n
    Content-Type: application/json; charset=utf-8\r\n
    Accept-Encoding: gzip\r\n
    \r\n
    [Full request URI: http://############.duckdns.org/]
    [HTTP request 1/1]
    File Data: 307 bytes
JavaScript Object Notation: application/json
    Object
        Member: _type
            [Path with value: /_type:location]
            [Member with value: _type:location]
            String value: location
            Key: _type
            [Path: /_type]

....

        Member: batt
            [Path with value: /batt:91]
            [Member with value: batt:91]
            Number value: 91
            Key: batt
            [Path: /batt]
...

        Member: lat
...            
        Member: lon
....   
     Member: m
            [Path with value: /m:1]
            [Member with value: m:1]
            Number value: 1
            Key: m
            [Path: /m]
        Member: topic
            [Path with value: /topic:owntracks/hotzenplotz/hotzenplotz]
            [Member with value: topic:owntracks/hotzenplotz/hotzenplotz]
            String value: owntracks/hotzenplotz/hotzenplotz
            Key: topic
            [Path: /topic]
        Member: tst
            [Path with value: /tst:1709644633]
            [Member with value: tst:1709644633]
            Number value: 1709644633
            Key: tst
            [Path: /tst]
        Member: vac
            [Path with value: /vac:0]
            [Member with value: vac:0]
            Number value: 0
            Key: vac
            [Path: /vac]
        Member: vel
            [Path with value: /vel:0]
            [Member with value: vel:0]
            Number value: 0
            Key: vel
            [Path: /vel]

and here is the response which obviously my android client does not like

    HTTP/1.1 200 OK\r\n
        [Expert Info (Chat/Sequence): HTTP/1.1 200 OK\r\n]
            [HTTP/1.1 200 OK\r\n]
            [Severity level: Chat]
            [Group: Sequence]
        Response Version: HTTP/1.1
        Status Code: 200
        [Status Code Description: OK]
        Response Phrase: OK
    Date: Tue, 05 Mar 2024 13:17:19 GMT\r\n
    Last-Modified: Tue, 06 Feb 2024 13:44:05 GMT\r\n
    Etag: "65c237a5.3207"\r\n
    Content-Type: text/html\r\n
    Content-Length: 3207\r\n
        [Content length: 3207]
    Connection: close\r\n
    Accept-Ranges: bytes\r\n
    \r\n
    [HTTP response 1/1]
    [Time since request: 0.000113832 seconds]
    [Request in frame: 40]
    [Request URI: http://########.duckdns.org/]
    File Data: 3207 bytes
Line-based text data: text/html (99 lines)
    <!DOCTYPE html>\n
    <html lang="en-US">\n
      <head>\n
        <meta charset="UTF-8">\n
        <title>Recorder</title>\n
        <meta name="viewport" content="width=device-width, initial-scale=1.0">\n
        <meta name="mobile-web-app-capable" content="yes">\n
        <meta name="apple-mobile-web-app-capable" content="yes">\n
        <link rel="icon" sizes="192x192" href="static/recorder.png">\n
        <link rel="apple-touch-icon" href="static/recorder.png">\n
    \n
        <style>\n
          body, a, a:visited {\n
            font-family: Helvetica, sans-serif;\n
    \t      color: #3f72b5;\n
          }\n
          #usertable {\n
            font-family: Helvetica, sans-serif;\n
            width: 80%;\n
            border-collapse: collapse;\n
          }\n
    \n
          #usertable td, #usertable th {\n
            font-size: 1em;\n
            border: 1px solid #305e9f;\n
            padding: 3px 7px 2px 7px;\n
          }\n
    \n
          #usertable th {\n
            font-size: 1.1em;\n
            text-align: left;\n
            padding-top: 5px;\n
            padding-bottom: 4px;\n
            background-color: #305e9f;\n
    \t      color: white;\n
          }\n
        </style>\n
    \n
        <script type="module">\n
          \n
          import { debug } from "./utils/debug.js";\n
          import { fetchApiData } from "./utils/network.js";\n
          import { escapeHTML } from "./utils/misc.js";\n
    \n
          const hours = 60 * 60 * 1000;\n
          const days = 24 * hours;\n
    \n
          const f0 = new Date().toISOString();\n
          const f12h = new Date(Date.now() - 12 * hours).toISOString();\n
          const f7d = new Date(Date.now() - 7 * days).toISOString();\n
          const f30d = new Date(Date.now() - 24 * days).toISOString();\n
    \n
          const data = await( fetchApiData({ endpoint: "last", includeSearchParams: true, rootLevel: true }));\n
    \n
          for (const d of data) {\n
            if (!d._type) {\n
              debug("Skipping unknown entry:", d);\n
              continue;\n
            }\n
            if (d.username == 'ping' && d.device == 'ping') {\n
              debug("Skipping ping entry:", d);\n
              continue;\n
            }\n
            d.udev = `user=${ encodeURIComponent(d.username) }&device=${ encodeURIComponent(d.device) }`;\n
    \n
            const row = `<tr><td>${ escapeHTML `${ d.username } / ${ d.device }` }</td>\n
            <td><a href='map/index.html?from=${ f12h }&to=${ f0 }&format=geojson&${ d.udev }'>12h</a></td>\n
            <td><a href='map/index.html?from=${ f7d }&to=${ f0 }&format=geojson&${ d.udev }'>7d</a></td>\n
    \n
            <td><a href='map/index.html?from=${ f12h }&to=${ f0 }&format=linestring&${ d.udev }'>12h</a></td>\n
            <td><a href='map/index.html?from=${ f7d }&to=${ f0 }&format=linestring&${ d.udev }'>7d</a></td>\n
            <td><a href='map/index.html?from=${ f30d }&to=${ f0 }&format=linestring&${ d.udev }'>30d</a></td>\n
            </tr>`;\n
    \n
            document.querySelector('#usertable > tbody:last-child').insertAdjacentHTML("beforeend", row);\n
          }\n
    \n
        </script>\n
      </head>\n
      <body>\n
        <div>\n
          <ul>\n
            <li><a href="last/index.html">Live MAP of current locations</a></li>\n
            <li><a href="table/index.html">TABLE of current locations</a></li>\n
          </ul>\n
        </div>\n
    \n
        <div id='content'>\n
        <table id='usertable'>\n
          <thead>\n
            <tr><th>User/Device</th><th>Points</th><th>Points</th><th>Track</th><th>Track</th><th>Track</th></tr>\n
          </thead>\n
          <tbody>\n
          </tbody>\n
        </table>\n
        </div>\n
    \n
      </body>\n
    </html>\n
jpmens commented 8 months ago

The JSON from the device to the Recorder looks reasonable, but the response is an HTML document which is definitely wrong.

POST / HTTP/1.1\r\n Request Method: POST Request URI: /

I don't think that post is hitting the /pub endpoint, at least that's not what it looks like.

When you run ot-recorder on the console, or check the docker logs, do you see a location being displayed on the console/logs?

jpmens commented 8 months ago

To clarify: your app should be configured to POST to https://example.com/owntracks/pub

MrColumbo commented 8 months ago

oh - great - now it is working. I was not aware that i should setup the client with /pub

jpmens commented 8 months ago

oh - great - now it is working.

Glad you got it going.