marcone / teslausb

A smart USB drive for Tesla Dashcam - extended storage, auto archive, web viewer
MIT License
1.93k stars 354 forks source link

V10 - Changes to Sentry Mode #260

Closed bladec59 closed 5 years ago

bladec59 commented 5 years ago

From early view of release notes it looks like Tesla is changing/adding a folder to store Sentry Mode clips:

“Sentry Mode video clips are now saved in a separate folder on your USB drive to make them easier to review and manage. Also, the older Sentry mode video clips will now be automatically deleted if there is not enough space on the USB drive and Sentry Mode clips are using more than 5 Gigabytes of space.”

Not sure this impacts teslausb code or not.

fatfurrycat commented 5 years ago

Given that a lot of people seem to be having issues with slow write speed errors, it might be worth collecting data on which combinations work, and which don't. So, here's my config, which I've been using for the last 4 days (and over 600 miles of driving), where I'm not seeing any errors:

RHD UK Model 3 Performance Tesla software version: 2019.32.11.1 d39e85a Raspberry Pi Zero W (with PiHut CPU heatsink) Transcend 256GB micro SD card (https://www.amazon.co.uk/Transcend-256GB-microSDXC-Memory-Adapter/dp/B07N31KRZL) Installed from teslausb-buster-20190922 image Pi Zero is plugged into right hand USB port

For my install, I did the following: 1) Using Windows diskpart, I selected the SD card and then ran a 'clean' command to erase all partition data. 2) Put teslausb-buster-20190922 image on the card using Etcher. 3) Used the following config file details:

# Default variables for CIFS (Windows/Mac file sharing) setup
export ARCHIVE_SYSTEM=cifs
export archiveserver=192.168.0.100
export sharename=teslacam
export shareuser=teslacam
export sharepassword=password
export cifs_version="1.0"
# camsize of 14% gives 32GB image file size
export camsize=14%

# if you set camsize to less than 100%, but don't want music to use
# all the remaining space, you can specify how much space you want
# music to use by setting this variable
export musicsize=128G

# If you want to automatically copy music from a CIFS share every time
# the Pi connects to wifi, set the following variable. The share is
# assumed to exist on the same server as the archive share. It can
# be the same share as the share used for backing up recordings, but
# the folder needs to be different.
#export musicsharename=media/Music/Albums

# Wifi setup information. Note that Raspberry Pi Zero W only supports 2.4 GHz wifi
export SSID='MYWIFI'
export WIFIPASS='mywifipass'

# If doing a headless (i.e. automatic) setup
export HEADLESS_SETUP=true

# uncomment to enable a samba server that exports the recordings
#export SAMBA_ENABLED=true
# uncomment to enable guest access to the samba server
#export SAMBA_GUEST=true

# uncomment to enable teslausb to act as a wifi access point with the given
# SSID, so you can access it while on the road.
export AP_SSID='TESLAUSB WIFI'
# Change this! The setup script will not accept the default 'password'
# Also note that the wifi password must be at least 8 characters.
export AP_PASS='teslausbpass'
# IP address is optional. The AP will give its clients IP addresses in the
# x.x.x.100-150 range so make sure you don't put the AP's IP address in
# that range.
#export AP_IP='192.168.66.1'

The resulting software was pulled from the main-dev branch.

Following the completion of the headless setup, I also ran: sudo -i /root/bin/remountfs_rw /root/bin/setup-teslausb selfupdate /root/bin/setup-teslausb reboot

This appears to be unneccessary, as no new packages were pulled down.

One thing I'm wondering is if the Pi could be thermal throttling. I have a heatsink on my CPU, which might be why I'm not seeing any issues. Monitoring temperatures and CPU clock speed might be an idea for those people who are seeing issues. From what I can see, the Pi Zero is slightly borderline in terms of speed when it comes to emulating USB storage. Any CPU speed reduction might well drop performance below the threshold that the car requires.

Some possible commands to try would be:

root@teslausb:~# vcgencmd measure_clock arm && vcgencmd measure_temp && vcgencmd get_throttled 
frequency(45)=700000000
temp=39.5'C
throttled=0x0
root@teslausb:~# vcgencmd measure_clock arm && vcgencmd measure_temp && vcgencmd get_throttled 
frequency(45)=1000000000
temp=40.6'C
throttled=0x0

or possibly something like this:

while true; do echo `date` && vcgencmd measure_temp && vcgencmd measure_clock arm && vcgencmd measure_clock core; vcgencmd get_throttled; sleep 10; done >> /var/tmp/throttled.log

My clock frequency seems to cycle between 700MHz and 1GHz, dependent on CPU load. For reference, 'throttled' can take the following values:

What:       /sys/devices/platform/soc/soc:firmware/get_throttled
Description:    This file returns the firmware get_throttled value which is 
        divided into 2 parts:
        - lower 16-bits are the live value
        - upper 16-bits are the history or sticky value

        Bits:
        0: under-voltage
        1: arm frequency capped
        2: currently throttled 
        16: under-voltage has occurred
        17: arm frequency capped has occurred
        18: throttling has occurred

        This file can be polled for changes.

What:       /sys/devices/platform/soc/soc:firmware/get_throttled_mask
Description:    This file does two things:
        1. Describes which bits should denote a change when polling
        2. The upper part is used when clearing the sticky bits

        Default value is 1 << 16.

What:       /sys/devices/platform/soc/soc:firmware/get_throttled_poll_ms
Description:    Specifies the get_throttled polling interval in milliseconds 
        (default 2000ms). The kernel polls this value to detect 
        under-voltage and to support polling the get_throttled file for 
        changes. Setting this to zero disables polling.
marcone commented 5 years ago

# camsize of 14% gives 32GB image file size export camsize=14%

Protip: you can just do export camsize="32G" to get a 32GB image.

One thing I'm wondering is if the Pi could be thermal throttling.

I doubt it, because the Pi doesn't use that much CPU when running teslausb, but this could be checked by fixing the CPU at a low speed and running speed tests against the CAM drive from an attached computer. Thanks for sharing your setup.

andrewgarfield commented 5 years ago

Just wanted to write back in case anyone is following this thread.

I have been using my rpi 4 as the dashcam for a couple days now and on my way to work today I got the error message that my storage wasn't fast enough. :( So, my plan now is to buy a fast write speed memory card and retry it with the raspberry pi zero. I would far prefer to use the rpi zero over the rpi 4. If this doesn't work, i'll try this new memory card with the rpi 4.

vita10gy commented 5 years ago

I had the issue with a heatsink too. I posted in the other thread, but I think I fixed it by re-flashing. I left 20% of the drive for teslausb this time and I didn't enable samba.

It complained about the speed before I ever updated it though, so I'm not sure what the issue was.

Maybe v10 just wants a fresh start.

fatfurrycat commented 5 years ago

camsize of 14% gives 32GB image file size export camsize=14%

Protip: you can just do export camsize="32G" to get a 32GB image.

Thanks @marcone. I wondered if it would work the same as the music partition. Good to know.

One thing I'm wondering is if the Pi could be thermal throttling.

I doubt it, because the Pi doesn't use that much CPU when running teslausb, but this could be checked by fixing the CPU at a low speed and running speed tests against the CAM drive from an attached computer. Thanks for sharing your setup.

No problem. If I get time, I'll try pegging the CPU speed to see if it makes any difference.

miles267 commented 5 years ago

camsize of 14% gives 32GB image file size export camsize=14%

Protip: you can just do export camsize="32G" to get a 32GB image.

Is it necessary to put camsize value in quotes? like export camsize="32G" ? I currently have just as export camsize=32G and there weren't any errors on setup. I'm just confused when these quotes are necessary.

marcone commented 5 years ago

In this case they're not needed.

miles267 commented 5 years ago

Just encountered same Pi4 situation as @andrewgarfield. Worked perfectly for 2 days before reporting drive too slow. Almost 48 hours exactly. This issue just becomes increasingly peculiar. Am discouraged :-(

japhule commented 5 years ago

Had issues with slow write on 922 image with 128gb samsung pro endurance. The issue usually shows up after I leave the car in sentry mode on for several hours. I switched to the 615 image with 32gb reserved for dashcam and 16gb for music and using the nofua=1,1 option and haven't had issues yet in 2 days. Will continue to monitor.

acraigl commented 5 years ago

Had issues with slow write on 922 image with 128gb samsung pro endurance. The issue usually shows up after I leave the car in sentry mode on for several hours. I switched to the 615 image with 32gb reserved for dashcam and 16gb for music and using the nofua=1,1 option and haven't had issues yet in 2 days. Will continue to monitor.

What is the he nofua=1,1 option? I'? I'm not familiar with it. Is it just a line you add to the .conf file? What does it do?

japhule commented 5 years ago

People have mentioned using the nofua option in "USB Storage Slow" thread, so I went ahead and added it. I'm not sure if that's the main fix for my issue, but I'm guessing the fix is coming from limiting the size of the dashcam and music storage, with the extra space being used for additional recent clips storage.

In any case, the nofua modification would be to:

sudo -i /root/bin/remountfs_rw

Then modify the file:

nano /etc/modprobe.d/g_mass_storage.conf

And adding:

nofua=1,1

to the end of the file.

acraigl commented 5 years ago

For the first time since I got v10, I did not have a "too slow" warning. Using a Sandisk Extreme 200GB card, 100% allocated to CAM. The only change is I'm using the June Stretch build instead. I will continue to monitor over the next few days. One thing worth noting -- Only my sentry clips directory seems to get populated. No "drive" videos have been uploaded/sync'd since 9/30. Wonder what's up with that??

marcone commented 5 years ago

For the first time since I got v10, I did not have a "too slow" warning. Using a Sandisk Extreme 200GB card, 100% allocated to CAM. The only change is I'm using the June Stretch build instead. I will continue to monitor over the next few days. One thing worth noting -- Only my sentry clips directory seems to get populated. No "drive" videos have been uploaded/sync'd since 9/30. Wonder what's up with that??

What do you mean by "drive" videos ?

acraigl commented 5 years ago

I mean videos while in motion. Only sentry videos are getting synced... just the ones while I'm parked.

marcone commented 5 years ago

@acraigl teslausb has never archived the "Recent Clips" folder

acraigl commented 5 years ago

@acraigl teslausb has never archived the "Recent Clips" folder It's just that I used to get a "top-level" folder of a date, and now everything is in a sub-folder off of "SentryClips". If this is the new normal, I'm OK with it, but I just expected to see more date folders appear.

miles267 commented 5 years ago

@acraigl what steps did you use to setup the June Stretch image And v10?

I tried on a RP0 with a FAST 128 GB SanDisk Extreme Pro microSD card with camsize=100% and my .conf file without success. It never completes the setup, I get the /etc/rc.local compatibility error. When I run apt-get update and apt-get upgrade, it grabs and installs many updates. I can then run ./setup-teslausb, but it never manages to mount my /mnt/archive which buster image has no issue with.

acraigl commented 5 years ago

@acraigl what steps did you use to setup the June Stretch image And v10?

I tried on a RP0 with a FAST 128 GB SanDisk Extreme Pro microSD card with camsize=100% and my .conf file without success. It never completes the setup, I get the /etc/rc.local compatibility error. When I run apt-get update and apt-get upgrade, it grabs and installs many updates. I can then run ./setup-teslausb, but it never manages to mount my /mnt/archive which buster image has no issue with.

I'm using CIFS 3.0, if that makes a difference. My (redacted) conf is attached. teslausb_setup_variables_v10_working_stretch.txt

miles267 commented 5 years ago

I'm using CIFS 3.0, if that makes a difference. My (redacted) conf is attached. teslausb_setup_variables_v10_working_stretch.txt

Yes, I'm also using CIFS 3.0 as I was on buster image. Since I'm using the same config file from my buster image actually. The RP0 connects to wifi without issue, but there are a few nuances I wasn't sure whether you encountered a need to do the same sequence of commands?

1.) I image to Stretch June image and copy my config file 2.) after rebooting, the headless setup will fail on /etc/rc.local compatibility issue 3.) I then perform the sudo apt-update and apt-upgrade steps and there are many updates applied 4.) I'm then able to re-run the ./setup-teslausb script successfully (not rc.local issues)

Aside from my RP0 not connecting to my /mnt/archive (CIFS windows share), when I check the /mnt/cam, there isn't the same folder structure under /mnt/cam as there is with the Buster image.

japhule commented 5 years ago

@miles267 When I set up my RP0 over the weekend, I only needed to do step 1 for everything to work initially. Sounds like an issue with your config file. Can you post your full config?

miles267 commented 5 years ago

@miles267 When I set up my RP0 over the weekend, I only needed to do step 1 for everything to work initially. Sounds like an issue with your config file. Can you post your full config?

Thanks @japhule. Here is the conf file I'm attempting to use with the June Stretch release. It works fine, as is, with the latest Buster release.

teslausb_setup_variables - Copy.txt

japhule commented 5 years ago

@miles267 Your config seems fine. The main difference from what you have versus what I have is that I didn't set a timezone and I'm also using a different CIFS version (1.0) but that shouldn't matter much. I would try again without setting a timezone.

miles267 commented 5 years ago

For the first time since I got v10, I did not have a "too slow" warning. Using a Sandisk Extreme 200GB card, 100% allocated to CAM. The only change is I'm using the June Stretch build instead. I will continue to monitor over the next few days. One thing worth noting -- Only my sentry clips directory seems to get populated. No "drive" videos have been uploaded/sync'd since 9/30. Wonder what's up with that??

@acraigl is your setup still working as we hope? or have you encountered the drive too slow error since first posting this?

acraigl commented 5 years ago

@miles267 So far no error. I'd like to give it another day or two before declaring victory.

c-E-c-777 commented 5 years ago

I've had the buster image working for about a week now. I was using the master build but in the process of trying to get it working with v10 I followed gacevedo's steps and haven't had any problems since. No USB speed errors either. Only oddity is that music from the pi will occasionally pause and eventually give a loading error when trying to play/pause; skipping to the next/previous song resolves the issue (only a pain when listening to Fear Inoculum...).

th3fallen commented 5 years ago

i tried the buster image on a pi4 and during a 30m drive saw it disconnect (remove the dashcam icon) several times and each time was about 5m before it returned...

miles267 commented 5 years ago

So many Tesla owners reporting USB drive slow errors here: https://teslamotorsclub.com/tmc/threads/anyone-still-rocking-a-pi-zero-w-smart-drive-for-teslacam-on-sentry-mode.143582/page-14#post-4081176

Also, have just received a new update 2019.32.12.1 today with bugfixes. Will see whether perhaps it helps this issue?

miles267 commented 5 years ago

Installed Buster image on RP0 with a SanDisk Extreme Pro microSD card. Camsize set to 16GB, music at 4GB. CIFS 3.0. Have not used nofua=1,1 at this moment. So far, no slow drive errors after 1.5 days although the car is now on 2019.32.12.1. If anything, I do notice that my dashcam icon comes and goes. It's possible it was doing that with my Samsung T5 SSD USB drive, but now I'm just too focused on this teslausb project :-)

NOTE: though I suspect this new car software update contains bugfix, there's no confirmation of what they are exactly.

miles267 commented 5 years ago

So far I’m 2 days in with my config above. Fortunately no error so far. Hope it continues.

acraigl commented 5 years ago

@miles267 So far no error. I'd like to give it another day or two before declaring victory.

3rd day with about 10 hours of sentry mode enabled, no issues with slow writes reported. Reminder this is with the stretch image. My car just got notified to download software... waiting for the sync to complete and will run it and test again tomorrow.

But I think I'm ready. VICTORY!

japhule commented 5 years ago

Same here. I'm over 3 days now with no issues using the 615 image.

miles267 commented 5 years ago

Just a general question - when should the Dashcam icon appear and when shouldn’t it? Am trying to determine whether I have a teslausb issue or if it not appearing while parked at home is normal.

sottile commented 5 years ago

If you turn sentry mode OFF at home then the USB port is powered off after about 5 minutes. When that happens the pi is off and the icon goes away.

Chewed up and spit out by autocorrect on my iPhone

On Oct 11, 2019, at 12:40 PM, miles267 notifications@github.com wrote:

 Just a general question - when should the Dashcam icon appear and when shouldn’t it? Am trying to determine whether I have a teslausb issue or if it not appearing while parked at home is normal.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

marcone commented 5 years ago

If you turn sentry mode OFF at home then the USB port is powered off after about 5 minutes. When that happens the pi is off and the icon goes away.

Presumably the screen will also be off in that case though :)

The icon should show whenever the Pi is fully booted and not archiving.

sottile commented 5 years ago

hah. yeah, right. I should have said if you start “fiddling” by going in and out of the car causing it the pi stop and then reboot. AKA “impatient user” :-)

I have noticed that it does the a while (3-4 minutes) for the full reboot. And yeah, if you are doing things like archiving (or pulling from the music archive) it’ takes a while to get back to USB mode. I have noticed that I’ve had to “help” it as the led aren’t flashing the way I want. Could be my USB hub in there interfering.

We need a little multi-processing here to speed some things up :-)

On Oct 11, 2019, at 4:06 PM, marcone notifications@github.com wrote:

If you turn sentry mode OFF at home then the USB port is powered off after about 5 minutes. When that happens the pi is off and the icon goes away.

Presumably the screen will also be off in that case though :)

The icon should show whenever the Pi is fully booted and not archiving.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/marcone/teslausb/issues/260?email_source=notifications&email_token=ABH3N75LXL4VWCH5HTHPAL3QODMDZA5CNFSM4IW3ZYZKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBBCM2I#issuecomment-541206121, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABH3N7Y6KIBL5MJCDOVKRBDQODMDZANCNFSM4IW3ZYZA.

jbienz commented 5 years ago

In other words, even though the car was running V10 for two weeks and recording rear camera footage, there were no "too slow" problems during that time.

The "too slow" problems began after I updated to the latest version of teslausb.

To me, this suggests that the problem is not because of having an additional camera feed. That additional camera feed was being written from 9/27 to 10/9. I have Sentry Mode turned off at home, so sentry was not trying to write while a copy was in progress. (And no files were being copied during that time.)

I would really appreciate if someone could help answer the following:

  1. What is the Buster Build? I'm not finding much info on this.
  2. What is the June Stretch Image?
  3. Are people suggesting that we should have Sentry on at home to fix the "too slow" warning? Even if we already have the API keeping the car awake?

Thanks everyone. I really want to keep teslausb in the car but I don't want to lose recording. When "too slow" happens, it looks like all recordings shut down.

Really appreciate the help.

acraigl commented 5 years ago

@miles267 So far no error. I'd like to give it another day or two before declaring victory.

3rd day with about 10 hours of sentry mode enabled, no issues with slow writes reported. Reminder this is with the stretch image. My car just got notified to download software... waiting for the sync to complete and will run it and test again tomorrow.

But I think I'm ready. VICTORY!

Well, crap. :( Everything was great until I got 2019.32.12.2 58f3b76. Went 3 days with 10+ hours of sentry mode and with this version I'm almost immediately getting the "too slow" warnings. I've switched (temporarily) to a USB thumbdrive, much to my chagrin, as I don't want to miss any potentially important videos. But I'll keep experimenting with the builds that come out. Looking forward to folks reporting their observations. I'll continue to report mine as well

miles267 commented 5 years ago

@acraigl I just had same thing happen to me with my SanDisk Extreme 256gb card with June Stretch image and camsize 100%. Am going to swap back in my Buster image with SanDisk Extreme Pro camsize 16GB and 4GB music that was working to see whether it’s a matter of too much camsize.

acraigl commented 5 years ago

@acraigl I just had same thing happen to me with my SanDisk Extreme 256gb card with June Stretch image and camsize 100%. Am going to swap back in my Buster image with SanDisk Extreme Pro camsize 16GB that was working to see whether it’s a matter of too much camsize.

Good deal. Let us know!

japhule commented 5 years ago

I updated to 12.2 on Friday evening and haven't seen any slow write issues yet. On Saturday evening, sentry mode captured 11 events in a 5 hour period. This is with Samsung Pro Endurance 128gb on the 615 image. Will continue to monitor...

acraigl commented 5 years ago

Is there any understanding at this point if Endurance vs Speed is the better choice?

The endurance cards are generally rated at transfers up to 100MB/s, but so is the card I'm using now, Sandisk Ultra, and it reports too slow. The high-speed is rated at 170MB/s. I guess I'm asking if I had to buy (another) SDcard, which is the better option? Or does it not matter as long as it's a class 3?

@japhule Is this the card you're using? It's a class 1. Hmmm https://www.amazon.com/Samsung-Endurance-128GB-Micro-Adapter/dp/B07B984HJ5/

japhule commented 5 years ago

@acraigl That is the card that I'm using.

Class 1 is more than enough speed for the dashcam recordings. Each 1 minute clip is about 30 MB, which is equivalent to 500 KB/s. With 4 cameras, it's writing at 2 MB/s. My understanding is that the RPi0's emulation of the storage drops the write speed significantly but should be enough still for recordings.

Endurance cards are supposed to be more ideal for security/dashcam applications where there is constant writing AND deletions to the media. I think they have better memory controllers that account for this. This was a bigger issue with smaller cards previously since there is less space to write to and they filled up completely quickly, which would lessen the life of the media quicker. With larger cards, they don't fill up as quickly and last longer, but I'd still prefer endurance over ultra high speed. I think Class 3 cards are meant for applications like general 4k video recording (not dash/security) where there is not constant writing AND deletions happening.

Here's a site that tests various cards for dash cam usage. https://carcamcentral.com/guide/recommended-microsd-cards-for-dash-cameras

acraigl commented 5 years ago

@japhule Thanks for the info. I guess I'm just not clear on why the endurance card is going to work better than what I have since that's the speed mine works at. But since it's working for you, I suppose it's worth a shot.

jbienz commented 5 years ago

Can someone please let me know what Buster Build is and what the June Stretch Image is?

@japhule How can you tell what version you have installed? Does it show up somewhere during an auto-update?

@acraigl The main reason you want high endurance is because they are designed to handle many deletes and rewrites. Non-endurance cards will run out of write operations sooner and become read-only cards. High endurance cards are commonly used in security cameras and dash cams.

marcone commented 5 years ago

Can someone please let me know what Buster Build is and what the June Stretch Image is?

They are the prebuilt flashable images referred to in the installation instructions

jbienz commented 5 years ago

They are the prebuilt flashable images referred to in the installation instructions

Thank you @marcone. Sorry I didn't notice these names before in the releases. I've recently just been doing selfupdate.

What does @japhule mean by "I updated to 12.2"? Looking at the releases I only see version numbers like 1.2, 1.2a and 1.3.

japhule commented 5 years ago

@jbienz 12.2 is the car firmware

jbienz commented 5 years ago

@jbienz 12.2 is the car firmware Oh. I see. I've had 2019.32.12.2 for 4 days now and it didn't make a difference for me. I'm going to try the vm.* caching settings from the other "too slow" issue and start following that thread. Thanks for all the help.

miles267 commented 5 years ago

@jbienz what vm caching settings?

jbienz commented 5 years ago

I completely rebuilt my Pi last night as described in this post.

After the rebuild, I've noticed that Tesla seems to have only created two folders in the /mnt/cam folder:

RecentClips and SentryClips

I do not see any SavedClips folder as is mentioned in archiveloop line 279 and line 280.

Line 279 and 280 also do not do anything with the RecentClips folder.

@sottile said something in his message on 9/17 about ignoring RecentClips, but why?

I finally have teslausb working again, but none of the videos are ending up on my server because they're all in the RecentClips folder.

This seems like a bug. What am I missing here?