Permission issue -- testing Quicksetup

[whmoorejr]

I'm running MQTT and owntracks on the same raspberry pi. I have everything working pretty well except for a couple oddities with having a few older phones in my setup (running older versions of owntracks). It still works, but can't do certain things like enable them to receive commands to force a location update. (Seperate issue but not a show stopper)

On the server side, I want to be able to have more control over manipulating data. The data store is located in the default location which works, but I started seeing issues when I tried to delete a user. During testing I created a user with a typo in the name. Re-created a user on that device, but the old user is still there.

I tried the "Kill" command but got the "No comprendo" message. Doing a version check, however, shows that "WITH_KILL = yes" so ? deleting a file is combersome as I can't use cyberduck or GUI due to permissions. I have to telnet in and use sudo commands to delete stuff in the var/spool/owntracks/recorder/store/rec/(user_directory)/(YYYY-MM.rec).

When I finally deleted the user file and folder under rec directory, the user is still visible on devices and on the recorder page.

I also tried getting around the permissions thing by moving the store to home/me/Documents/owntracks...../store (I made a full copy at that location).... then I get an error with owntracks... something about ghash Permissions Denied.

So I know I messed up something somewhere but I'm not sure what direction to go in now.

[jpmens]

No worries, nothing is 'messed up'. :-)

You are seeing "old" users on your devices because they have retained messages on the MQTT broker, most likely. There are two things you'll want to do to rid yourself of those:

1. publish a retained NULL payload to the topic of those users, something like

   mosquitto_pub -h server .... -t "owntracks/olduser/olddevice" -r -n

   the -n means null and -r -s retained. This will remove the messages from the broker and ought to also clear them from your phone.

2. if they're not cleared from your phone, go to the friends list and swipe the friend away.

The second issue is ridding yourself of old .rec files. The kill command is only sometimes compiled into our Recorder and might actually be deprecated, though based on what you report and the fact you can't easily access the files, we might actually have to reconsider.

Be that as it may, to remove historic files for a particular user, you change into the Recorder's spool directory (typically /var/spool/owntracks/recorder/ and remove

• rec/olduser/olddevice
• cards/olduser/olddevice
• waypoints/olduser/olddevice
• etc.

It's not easily possible to move the spool directory away; the Recorder (ot-recorder) is a SUID program which will re-create files as user owntracks. You could get around that, but I fear that an upgrade would break things for you, so I'd prefer not explaining how that works ...

Let us know if this info has helped.

[jpmens]

I'll also advertise our new quicksetup system which you might like to look at.

[whmoorejr]

I love the look of the new quicksetup. I do already have a couple questions... If I follow allong with the quicksetup, will it overwrite my existing owntracks setup or should I uninstall owntracks first?

I'm only using this raspberry for MQTT & Owntracks, would it be easier to start with a clean install of Raspbian 12 (bookworm)? Is the mosquito included in the setup now? I thought I installed it seperately before installing owntracks. (following an example in the owntracks guide if I remember correctly)

If I start from scratch with a fresh install of Raspbian, should I set it up as "me" or should I set it up as the user "owntracks" with a password to avoid permission issues later? Any tips or suggestions here would be appreciated as well. This is an "onwtracks" dedicated box, so I don't need my own login. Is there a preferred raspbian OS? bookwork 64bit lite or full with desktop? or 32bit with no desktop?

Maybe worth considering a pre-configured image file of a raspian setup. Then just a couple steps to sudo nano the configuration files for MQTT and ot-recorder to make it user specific and less trouble shooting to figure out which step the user skipped or messed up.

Backstory: I landed here when the unofficial API for life360 was killed by life360. As an alternative, I think Owntracks is superior and I wish I knew about it before. Once I have owntracks figured out and working on a couple test devices, then I'll start figuring out how to incorporate it into my home automation platform (It's a mac based platform, Indigo).

[jpmens]

I'm only using this raspberry for MQTT & Owntracks, would it be easier to start with a clean install of Raspbian 12 (bookworm)?

Definitely, ideally you'd swap out boot drives to make sure we don't break anything you need.

If I start from scratch with a fresh install of Raspbian, should I set it up as "me"

Go ahead and set up a "me". Our quickinstall will be run as root (sudo) so it will do what it has to do with appropriatea permissions. It will be a dedicated OwnTracks box. Bookworm 64 bit would be fine, and we don't need a desktop: if you're comfortable SSH'in in, that's what I'd use.

Please be aware you're a test bunny: to our knowledge the first live user, so please expect a few rough edges, but we're here to help.

Also if you notice rough edges or errors/omissions in the documentation, we gladly take issues or even patches to fix!

[whmoorejr]

I'm your man (Or bunny in this case). I know just enough in all the above technologies to be awkwardly comfortable. My only linux experience has been with some limited tinkering with a couple raspberry pi machines. I have amlost zero ms windows knowledge, but I do know my way around a mac. I know enough phyton code to write basic "hello world" types of scripts, and my JS knowledge is none. I have a new batch of SD cards on the way and I'll start with a fresh setup and take notes of my progress. SD cards should be here tomorrow and I'll start right away on it.

[whmoorejr]

I am cataloguing my progress... I'm currently crashing on step 4 in the process.... `TASK [verify some requirements] ** fatal: [localhost]: FAILED! => { "assertion": "ansible_distribution_release in [ 'bookworm', 'jammy' ]", "changed": false, "evaluated_to": false, "msg": "Assertion failed" }

PLAY RECAP *** localhost : ok=2 changed=1 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 `

[jpmens]

Show me the content of the file sys.info in the quicksetup directory, please. It ought to contain the name of the distribution you are using.

[jpmens]

We put that assertion in to make sure we're running on a distro known to us. @ckrey reports he used this:

[whmoorejr]

That's the same distro I used....

sys.info:

system info

Last bootstrap: 2024-02-19T03:49:24Z

Ansible version: 2.15.9 OS distro: Debian / 11 OS distribution: bullseye send_welcome: y

[whmoorejr]

Full writeup of my progress / steps so far:

Notes from a test Bunny:

Goal: Create a dedicated OwnTracks box on a dedicated rasperry pi. Most of the work will be done from a Mac… but the SSH stuff should be universal.

Hardware: Rasperry Pi 3 B+ https://www.amazon.com/gp/product/B07P4LSDYV/ Case, Fan, Power https://www.amazon.com/gp/product/B07BTHNW9W

Mac mini (To set up the Raspberry Pi) Software: Mac OS 14.2.1 Raspberry Pi Imager v1.8.4 Cyberduck (optional. Provides a visual folder view of the device you SSH into) MQTT Eplorer (optional. Handy for checking the trafic of the MQTT broker, seeing the layers of data being transmitted, etc.)

User Equipment: iPhone 6 x 3 iPhone 8 x 1 iPhone 10 x 1 iPhone 13 x 3

Note: For testing, I’ll use an iPhone 13 and an iPhone 6 to see the differences between old and new.

As of this writting, iPhone 6 and older are no longer supported by apple for udpates and there are some software differences as it will run an older version of owntracks. Some of the newer features aren’t supported on the older application (that I could tell).

Started with a 32GB Micro SD Card (A junk one to get going, around $7 USD) https://www.amazon.com/gp/product/B07R8GVGN9/

Once I have a stable platform, I’ll duplicate it to a card rated for longevity (Samsung 32 Pro Endurance $24 USD) https://www.amazon.com/gp/product/B07B98GXQT/

Using Raspberry Pi Imager, I followed the steps to install: Raspberry Pi OS (Other) > :Raspberry Pi OS Lite (64-bit) Debian Bookworm with no desktop environment

Note: I figure that going in from another computer (SSH) is what I’ll primarily do, since I won’t dedicate a monitor to a pi and it will just run headless, why add any extra bulk with a desktop configuration. Should also minimize anything that could go wrong later.

Pi Imaging Notes: Start with a new SD Card. Select the card in the software and Erase / Format as FAT32 Select your Raspberry Pi device (Pi 3 for myself) Select the os “Raspberry Pi OS Lite (64-bit) Debian Bookworm with no desktop environment (It’s under the menu option of "Raspberry Pi OS (Other) >" )

Apply OS customisation setting? Edit Settings: Set your user name and password. Configure wireless lan (if needed) Important, on the top, select “Services” and check “Enable SSH” (So you can access the pi from another computer to finish setting up or to mess with later. Click “Save” when done. Now Click “Yes” to apply. You may have to enter your Mac password to continue. The application will now write the operating system to the SD Card. Once done, it will eject itself from your computer. Stick it in the rasperry Pi and boot up. QuickSetup Notes: Below are just the actions I took and my own notes. More detailed instructions are in the online guide. Owntracks Guide: Main Site: https://owntracks.org/ Booklet: https://owntracks.org/booklet/ Quicksetup: https://owntracks.org/booklet/guide/quicksetup/ Step 1: good to go. Step 2: Consider adding the open cage website to the yaml file “opencagedata.com”. If you login with your github account, it will automatically generate an API key and email it to you. Once you refresh your screen, you will see your free API key on opencagedata.com as well. Step 3:

[jpmens]

OS distro: Debian / 11
OS distribution: bullseye

@whmoorejr you appear to be running bullseye (version 11) and not bookworm (version 12). Can you upgrade your OS easily?

[jpmens]

BTW, I hope I did not offend you when I jokingly said test bunny; I certainly didn't mean to! :-)

[jpmens]

One warning please, which isn't mentioned anywhere yet: when you get to configure the bootstrap process in configuration.yaml please specify distinct usernames for each device. I read above that you'll be configuring different phones -- they must (currently) be assigned to different users, so for instance

{ tid: w6, username: whmoorejr, devicename: iphone6 },
{ tid: w3, username: whmoorejr1, devicename: iphone-13},

etc. Note the different values for username.

[whmoorejr]

Absolutely not. I'm not a programmer by trade, but I love to tinker with stuff so I dabble is many different platforms. I have no expertise in any of these disciplines which means most of the time, I'm stuck begging for help from others that are more knowelegable. Having a chance to do something that might actually help others I think is an honor. I'm more than happy to fill the roll. I also get a kick out of the differences. Here we would jokinly refer to the subject as a guinnie pig. It doesn't really make sense since universally testing is more commonly done with mice or rabbits.

[jpmens]

Please ask for as much help as you need (or as we can give :-) You're helping us to improve on the software / documentation!

Please also run a git pull from within your quicksetup/ directory again.

[whmoorejr]

Interesting. So if User A has 3 devices, the device names can all be the same, but tid and username should all be different. With my current yaml file, I have unique tid and usernames, but all the device names I set to "myphone":

{ tid: AH, username: anna, devicename: myphone },
{ tid: WM, username: william, devicename: myphone},
{ tid: VM, username: vikki, devicename: myphone},

... etc.

[jpmens]

Your configuration looks fine. (I took the liberty of editing your comment to add a line with three backticks each before and after; code looks neater that way.)

Note, that this ought to be temporary only, but we're looking how to fix/implement identical usernames for the future. The issue regards password generation per user.

[whmoorejr]

I started to update the operating system.... then I thought, it's basically blank, why don't I just start over. At the same time, I noticed a newer version of Raspberry Pi Imager was available (Version 1.8.5) <- Which doesn't make a differece from 1.8.4.

What I did notice, if you select operating system first, you will see "Raspberry Pi OS Lite (64-bit) Debian Bookworm" availiable for raspberry pi 3/4/5... BUT, if you select your Raspberry Pi Device as a 3, then Bookworm goes away and you are left with Bullseye. (Which I didn't catch the switch when I imaged the SD card before.)

This time, I left the "Select your Raspberry Pi Device" as blank or "No Filtering". From what I can tell, it's imaging a new SD card with bookworm. I'm at about 80% now. I'll finish this part up, go back to the quicksetup steps and report back shortly.

[jpmens]

Please run a (hopefully last) git pull in quicksetup/ directory before you run the bootstrap.

[ckrey]

I have a Pi 3+ too and installed the bookworm version above without problems

[whmoorejr]

I would suggest adding this prior to step 1....

$ hostnamectl Confirm that the Operating System: Debian GNU/Linux 12 (bookworm)

Went back through Steps 1-3. Lookinig good. Did a last git pull after editing the yaml file. "Already up to date"... so I guess that's good.

Step 4: $ sudo ./bootstrap.sh
result: Fatal on TASK [lego: enroll at letsencrypt]

I'm not positive... but I do see this in the log: "Timeout during connect (likely firewall problem)"

maybe a port forwarding issue? On my previous install, only MQTT broker had an open port (1883) ot-recorder port, (8083) was not open, but I only used it locally.... same with port 9001 for cards. I think only port 1883 is open on that LAN address.

[jpmens]

OK, we're getting somewhere...

You're right: we haven't specified which TCP ports you'll need. I assume you're behind a router but can open some TCP ports?

You will require the following, please, and do note we firewall the rest on your Raspi:

• 80 so that Let's Encrypt can connect to your server when enrolling an SSL certificate
• 443 to be able to access the Web interface from the Internet. If you do not wish to do so, leave this one closed, but you'll want it open from withing your home. You will not need 8083 as we do this all over 443 (HTTPS)
• 8883 for MQTT. We don't u se 1883 as that's insecure. The devices (your iPhones) will be configured to use encrypted port 8883.

Please re-run ./bootstrap.sh when you're ready.

[whmoorejr]

Opened 80, 443 and 8883. (closed 8083). Got past lets encrypt. Next up looks like a python package....

TASK [lego: get certificate information] *** fatal: [localhost]: FAILED! => {"changed": false, "msg": "Cannot detect any of the required Python libraries cryptography (>= 1.6)"}

[jpmens]

Ah, yes, that's likely due to our changing the installer. I hope you don't think we don't test this stuff, although i do admit it looks as though we don't test this stuff ... (sighs)

Could you please, on the command line run

$ sudo apt install python3-cryptography

and re-run ./bootstrap.sh

[whmoorejr]

Success! I ran bootstrap a second time after it succeeded.... mostly because it made me feel more accomplished to see all the green responses.

Next up looks like setting up the actual devices. My thought is to delete the current instance of owntracks from everything to make sure I'm starting from scratch and not adding any old data into the system. Then I will re-install owntracks and continue following the quicksetup.

My only delay is that the iPhone 10 is currently out of town with my daughter for a week. It is paired (same apple family) with the other iPhone 6 devices. I may need the 10 to do a work around to install software on the older phones. Apple is funny that way. But I will see how far I can get with the devices on hand and I'll keep notes.

[jpmens]

Excellent, congratulations!

You shouldn't need to delete the apps from your phones. Go ahead and login to your OwnTracks site (https://example.com/owntracks) with your user and password and click on the inline configuration link.

[whmoorejr]

At first login, is password the same as the userid? I didn't set a pasword for any of the users yet.

[jpmens]

As documented, the password was generated. You'll find it in a file named /usr/local/owntracks/userdata/<username>.pass

[whmoorejr]

Still making progress.... slowly, but steady. I've got one phone (mine) configured. I'm going to tinker with it a little bit and then start adding other phones.

[jpmens]

When you click here https://owntracks.org/booklet/guide/qs/rabbit-10663.png on "configure it with a click" your iPhone should offer to "open in owntracks" and that configures the app.

[whmoorejr]

Yes. That worked like a charm. I'm up to 3 phones. It's not actually taking me this long.... just crazy dad duties this weekend on top of my nerd projects. With a newer phone, the process works super simple. On an older version of owntracks (Version 13.1.7) on iOS 12.5.7 (iPhone 6). It's a little tricker. What's neat is I screwed up, but it fixed itself. When I clicked the "configure it with a click", it basically just wiped all the old data from the phone, changed the MQTT port and that's about it. So I manually entered the DNS, user and password and Viola! Connected.... but wrong... I forgot to change the tid. Changed that, and cool, done. Opps, forgot to change DeviceID from the random generated one to "myphone". Then from that iphone6, I deleted the two incorrect entries. On the other two phones, I only see the corrected entry. I haven't looked in the rasperry pi to see if it made devices for my two "oops" attempts.

So far it looks like you have been busy. I'm already seeing quite a few features and options that I don't remember before.

Question. I will be accessing the MQTT from another device outside of owntracks. In this case, it will be a server computer. To access the MQTT Broker, should I add that server in the same yaml file like it was a phone to generate a username and password? I think that would work, but then will server will show up on all the other devices as a "Friend"? Or will it only show up if my server publishes a location message? The server will also be subscibing to owntracks and maybe publishing. Publishing Example, I have a virtual button to request a location update. It will post to the topic owntracks/bill/myphone/cmd with the payload {"_type":"cmd", "action":"reportLocation"}

[whmoorejr]

A couple more questions, maybe a glitch?.... I currently have 3 devices set up. On the devices under friends I see all three devices just fine. On the frontend, I currenlty just get a blank map with no selection options... and it always starts off somewhere in the Atlantic off the coast of Africa. I plan on going for a drive tomorrow to see if my locations change if the frontend will change.

On the Device Table and Live Map, one of my "oops" devices is showing. Is there a Kill option? user indigo, device A33EED6F-BEAF-46AB-BA50-A811451B4594. If I need to delete stuff, that's ok.... just please list out all the items I need to sudo delete and such.

With the changeover, Is the API stuff gone?

Cards? Adding a card from the computer side... is this a work in progress? It looks like that might have changed from what I remember (I think there was a mini web server thing you could add in) Also I noticed the example card is missing (Quick Setup example card for jane.) I can add cards from my phone on the newer phones. The older phones don't have a create a card option, but it would be nice to be able to centrally manage it.

Sorry if all of this sounds negative.... quite the opposite. I love the simplicity of the new setup. Compaired to the previous install before the quicksetup, this was a piece of cake. From what I can tell, this method is a lot more secure than what I had before. Very nice!

[jpmens]

Yes. That worked like a charm. I'm up to 3 phones.

That sounds good.

On an older version of owntracks (Version 13.1.7) on iOS 12.5.7 (iPhone 6). It's a little tricker

To be expected; that's quite old ... I'm actually surprised the phone is still working. :)

Opps, forgot to change DeviceID from the random generated one to "myphone".

That wasn't random: you said earlier that that's the device name you gave it in configuration.yaml

To access the MQTT Broker, should I add that server in the same yaml file like it was a phone to generate a username and password? I think that would work, but then will server will show up on all the other devices as a "Friend"?

it will only show up if it publishes as a friend

It will post to the topic owntracks/bill/myphone/cmd with the payload {"_type":"cmd", "action":"reportLocation"}

Please note that's always been wobbly for iOS devices as they only report when they occasionally wake up.

On the frontend, I currenlty just get a blank map with no selection options... and it always starts off somewhere in the Atlantic off the coast of Africa. I plan on going for a drive tomorrow to see if my locations change if the frontend will change.

By 'frontend' I am assuming you mean our new Frontend with the blue bar across the top? (https://owntracks.org/booklet/guide/qs/rabbit-10665.png) Click on the hamburger menu above left if it's there. You can select users, dates to show etc. If the menu isn't there, you probably already have the fields across the top.

one of my "oops" devices is showing. Is there a Kill option?

assuming the user is actually called "indigo":

sudo rm -r /var/spool/owntracks/recorder/store/rec/indigo
sudo rm -r /var/spool/owntracks/recorder/store/last/indigo
mosquitto_pub -t "indigo/A33EED6F-BEAF-46AB-BA50-A811451B4594" -r -n -u _lr -P "$(cat cat /usr/local/owntracks/userdata/.lr.pw)

That's underscore lr (_lr) and dot lr.pw (.lr.pw)

With the changeover, Is the API stuff gone?

Definitely not. It is accessible via authentication (your OwnTracks user, your password) from "outside" if you have port 443 open, and from within your network, likewise on port 443. So the URL should be like https://example.com/owntracks/api/0/...

Adding a card from the computer side... is this a work in progress? It looks like that might have changed from what I remember (I think there was a mini web server thing you could add in)

Adding a card is still what it was: a retained publish to MQTT. Documentation is https://owntracks.org/booklet/features/card/

Compaired to the previous install before the quicksetup, this was a piece of cake

That was, indeed, the intention, and thank you very much for having pointed out some lacking in our documentation and a couple of wrinkles in the software, some of which I've already addressed.

Please feel free to continue reporting bugs and making suggestions, gladly at our quicksetup repo

[whmoorejr]

On the front end, I have the fields accross the top. Next to the person icon, the page defaults to "Show all" with a down arrow. When I click it, it doesn't show any of the users I've added so far. The download button on the right of the page opens up a window with an empty json {}. With the far right information icon, I get owntracks/frontend (2.12.0), but owntracks/recorder (Loading version...) <-- doesn't ever populate.

Following the quicksetup, I did verify that the Recorder is saving publised data $ tail /var/spool/owntracks/recorder/store/rec/bill/myphone/2024-02.rec

Side note, from some of your other posts, it looks like you are a mac guy.... have you ever used the home automation platform, "Indigo" ?

[jpmens]

Would you please refresh the Frontend page? It will need to re-acquire data from the Recorder API in order to populate users, etc.

Following the quicksetup, I did verify that the Recorder is saving publised data

Good.

You should also be able to access data via the API (URL above)

have you ever used the home automation platform, "Indigo" ?

I have not. I gave up all home automation experiments, of which I did a few, several years ago. I am a huge believer in physical switches for lamps, etc. -- they always work. :-)

[jpmens]

Do the other pages work for you?

• Device table
• Live map

[whmoorejr]

Frontend still isn't working for me. Device table and Live map are working fine. using this: https://192.168.1.86/owntracks/map/index.html?user=bill&device=myphone&format=geojson&from=2024-02-19 I did get the pretty map which showed me driving all over Houston today.

API is working for me as long as I get the formatting right. :-) working: https://192.168.1.86/owntracks/api/0/last?fields=tst,tid,addr,topic,isotst No Comprendo: https://192.168.1.86/owntracks/api/0/last[-d user=bill [-d device=myphone]] working: https://192.168.1.86/owntracks/api/0/last?user=bill&device=myphone (same as above, just formatted different)

But, yes... I can access all my data in one way or another. The frontend might be my go-to location to check device locations/history... but I can't figure out why it doesn't see the data.

I see where the frontend setup stuff is "quicksetup/files/frontend" Other than that, I'm not sure what to look for. Sorry I'm not much help.

Suggestion for Quicksetup.... maybe in the debugging section at the bottom.... add a few check things, like the lines to check that stuff is running properly: $ sudo systemctl satus mosquitto / ot-recorder / etc.

[jpmens]

The penny might be dropping ... I notice in the examples above, that you are using an IP address to access your site and API, so I am going to assume you are doing that to access Frontend as well?

That doesn't work.

The reason you configured dns_domain in configuration.yaml was so that Quicksetup would enroll an SSL certificate for you. Assuming that domain is bill.example, the only correct method of accessing your site is https://bill.example/.... You've had to use -k on curl(1), and you've had to add exceptions to your Web browsers for accessing https://192.168.1.68.

My question: but why? Why not use the URL with the domain name in it?

[jpmens]

In Safari you got a big fat warning:

[jpmens]

Should you for whichever reason (which would interest me) not be able to do so, you ought to be able to work around the issue with the following small trick.

Edit the /etc/hosts file on your Mac

$ sudo nano /etc/hosts

and add a line which looks like the following (bill.example being the domain name you configured in dns_domain):

192.168.1.86  bill.example

Back in the shell, a

$ ping -c2 bill.example

should show the 192.168.1.86 address.

Then please try to access https://bill.example/owntracks

[whmoorejr]

The penny might be dropping ... I notice in the examples above, that you are using an IP address to access your site and API, so I am going to assume you are doing that to access Frontend as well?

That doesn't work.

Weird habit, maybe? I would use the DNS address for an off-site connection, the URL for a LAN connection and localhost or 127.0.0.1 if I'm accessing from the same machine.

However, when I changed my url to use the DNS instead of the IP address, everything worked as expected.

That new frontend... absolutely beautiful. I love it! Simple, yet detailed and lightning fast. It would probably take 10 minutes or more to get that kind of information out an alternative platform, like Life360.

[jpmens]

Weird habit

don't do that again. ;)

That new frontend... absolutely beautiful

I'll let @linusg know you like it. So do I.

I'm closing this issue now, but please feel free to open a new one if you have questions or trouble. Thanks for testing!