USB Storage Slow?

[DonC52]

I have been wondering why the following occurs on my Windows 10 laptop and the Pi USB. I am using a 128G SanDisk Extreme Plus UHS so I would think it rather fast.

I put in an old "slow" USB stick into my Laptop port and copy a Sentry Mode directory, I get about 20MB/Sec transfer rate.

I put the Pi USB in with power applied from external source and copy the same directory, I get like 1.5MB/s to 2.5MB/s transfer rate.

Looking at top, there is a LOT of wait time going on. Is there anything that can be done to speed things up?

[CircuitGuy]

@dadealus I'm not quite buying that unless there's something else odd at play with the way Tesla is writing.

I have a Samsung Pro Endurance in both TeslaCam and a BlackVue DR900S-2ch 4k dashcam. The BlackVue writes far more data than Tesla, even with the v10 additional camera.

[dadealus]

@dadealus I'm not quite buying that unless there's something else odd at play with the way Tesla is writing.

I have a Samsung Pro Endurance in both TeslaCam and a BlackVue DR900S-2ch 4k dashcam. The BlackVue writes far more data than Tesla, even with the v10 additional camera.

BlackVue isnt using a containerized write on a Pi0. additionally MCU is constantly writing current camera operations to the drive as well as in parallel saving sentry clips in large batches from that camera bucket. So yes its possible over time BlackVue stores more. but if it never bursts up high enough for a brief moment to overfill the presumably very small cache it would never see that hard IOPS wall. And that wall might be for such a brief moment but the MCU might be hyper sensitive to waits from its storage.

I dont think I've seen a single person with a Samsung Pro Endurance not having issues on v10.

[ClanLee]

I also had issues with the Samsung Endurance Pro 64 GB, CAM set to 100% and Sandisk High Endurance 128 GB, CAM set to 100%.

I've since switched to Sandisk Extreme Pro 128 GB, CAM set to 65GB and Music set to 15GB, leaving around 38GB of "free space". I've been running the June release for almost three days without any issues.

[marcone]

not sure what the trigger is to empty the device to the smb share

It archives when it goes from "unable to reach the archive server" to "able to reach the archive server". So yes, if you leave it parked in wifi range with Sentry on, it will fill up.

additionally MCU is constantly writing current camera operations to the drive as well as in parallel saving sentry clips in large batches from that camera bucket.

I don't think this is the case. If you install iotop and run it while the car is on (doesn't matter if Sentry is on or not), you will see a burst of I/O every minute. This would seem to indicate that the car records to an internal buffer/disk, and then once it has finished one minute of recording, it moves the completed recordings to the USB drive.

[miles267]

I’m trying buster with SanDisk Extreme Pro 128GB card now. Camsize set to 16GB and Music set to 4GB. Also got the latest bugfixes 2019.32.12.1. Not sure whether any address the slow write speeds of the car.

I’ve tried repeatedly to install the June Stretch release without success. The config doesn’t run smoothly and never connects to my CIFS fileshare. Not quite sure what the issue is.

[infernix]

That's a nonstarter for me, as it would prevent snapshots from working, which in turn prevents extended recentclips and samba access from working.

You can use LVM to bypass the filesystem-on-filesystem performance hit that is implied. Would be quite a bit better than file with loopback. Not only that, you could snapshot the LVM LV and skip the entire copying of files as a means to create a snapshot. Might need some rethinking, I haven't looked hard at the snapshot code and its purpose. Saves space too I think?

(I also doubt that alignment is the issue here, given that people are reporting "usb too slow" using usb drives directly without teslausb)

You shouldn't just discard alignment issues like that. Wrong alignment can trigger two read-erase-write cycles on adjacent erase blocks on the physical card. This also ties into the other comment with using larger clusters for the FAT32 layout (-F 32 -S 4096). I havent looked at this in quite a few years, but IIRC SLC needs alignment on 512kb, MLC on 1024kb, TLC on 1536kb and QLC on 2048kb. A quick look at my sdcard shows:

Partition mmcblk0p1 alignment: SLC YES - MLC YES - TLC  NO - QLC YES - sector 8192
Partition mmcblk0p2 alignment: SLC YES - MLC YES - TLC YES - QLC YES - sector 540672
Partition mmcblk0p3 alignment: SLC YES - MLC YES - TLC YES - QLC YES - sector 4325376
Partition mmcblk0p4 alignment: SLC YES - MLC YES - TLC  NO - QLC  NO - sector 124835840

So p3 should be aligned, p4 likely not.

Furthermore, and I haven't tested this myself yet, there was someone in a reddit thread who claimed he had ext4 working instead of VFAT and all it took is to chmod 777 the TeslaCam dir. This might be faster than FAT32 from a kernel drivers' perspective on the Tesla side, so worth exploring as well.

[marcone]

Not only that, you could snapshot the LVM LV and skip the entire copying of files as a means to create a snapshot.

It's not copying files. It's using xfs as the filesystem for the backingfiles partition, so that "cp --reflink=always" can create a zero-overhead snapshot of the image.

[dadealus]

Is there a page detailing the reasoning for snapshots? Just trying to understand the inner workings of what happens when.

[marcone]

No, there isn't.

[wwwb0n3zcom]

Update: So it's been 3 solid days and still no errors. I have tons of events and even captured events while driving. Still not one single error. My configuration is above.

added link for ease: https://github.com/marcone/teslausb/issues/239#issuecomment-539257760

[dadealus]

Update: So it's been 3 solid days and still no errors. I have tons of events and even captured events while driving. Still not one single error. My configuration is above.

before you upgraded to the SanDisk what were you running before? Also I see you tried specifying 4096 blocksize correct?

[dadealus]

No, there isn't.

Is the purpose of the snapshot to hold a point in time to copy the data off the card and then purge the snap?

[marcone]

No, it's to keep more than an hour of "RecentClips" by taking a snapshot every hour, and to provide access to the recordings via Samba.

[dadealus]

No, it's to keep more than an hour of "RecentClips" by taking a snapshot every hour, and to provide access to the recordings via Samba.

how far back does it snap it? and when does it purge it?

[wwwb0n3zcom]

Update: So it's been 3 solid days and still no errors. I have tons of events and even captured events while driving. Still not one single error. My configuration is above.

before you upgraded to the SanDisk what were you running before? Also I see you tried specifying 4096 blocksize correct?

No, I didn't do the block size. I'm not sure how to add that into the file that's online.

Before I was using a SanDisk Ultra 200GB (100MB/s, C10, U1, Full HD, A1).

[dsgerbc]

So, I'm now getting this 'too slow' issue too, after the V10-compatible update. I'd like to go back to what I had, but not sure how far back I'd have to revert. Has anything major happened to the USB emulation over the past 1.5 months? I think I last updated in late August/early September, and it was working fairly well even on v10, minus the archiving part.

[marcone]

how far back does it snap it?

I don't understand what you mean by that.

and when does it purge it?

when the free available space on the backingfiles partition gets low (currently defined as "less than the size of the cam_disk.bin image") it starts deleting the oldest snapshots

[dsgerbc]

How does one disable this snapshot functionality?

[marcone]

One doesn't.

(well, I suppose you could put an exit at the top of /root/bin/make_snapshot.sh)

[natrlhy]

Hm no more discord server. Was going to ask this there. I have cam size set to 100% and wanted to try setting a hard value of say 80GB. Do I have to reflash and start over? This is for a 128GB SanDisk 128GB High Endurance on an RPi0

[marcone]

Discord server had to be recreated last week because of a snafu on Discord's end, but it still/again exists. See new invite link in the pinned issue.

Reflashing is the easiest way to change the disk size.

[dadealus]

how far back does it snap it?

I don't understand what you mean by that.

and when does it purge it?

when the free available space on the backingfiles partition gets low (currently defined as "less than the size of the cam_disk.bin image") it starts deleting the oldest snapshots

So if one has 32 GB SD card. and you set camsize to 15GB would the snapshots be contained and limited to the 15gb or would it save snapshots across the entire 32gb?

[marcone]

Your question doesn't really make sense to me in the context of how teslausb actually works. You should probably look at the code in make_snapshot.sh to see what it does.

[wwwb0n3zcom]

Hm no more discord server. Was going to ask this there. I have cam size set to 100% and wanted to try setting a hard value of say 80GB. Do I have to reflash and start over? This is for a 128GB SanDisk 128GB High Endurance on an RPi0

This is what I did to resize cam and music via telnet:

sudo -i
/root/bin/remountfs_rw
nano /root/teslausb_setup_variables.conf
umount /backingfiles/snapshots/snap*/mnt
rm -rf /backingfiles/*
/root/bin/setup-teslausb selfupdate
/root/bin/setup-teslausb
reboot

[infernix]

@marcone just had a colleague confirm that ext4 works for TeslaCam but not for music. He's using one ext4 partition for camera and a second FAT32 partition for music.

Was ext4 ever tried?

[natrlhy]

Thanks @wwwb0n3zcom I tried this but ended up reflashing to be sure.

My config now: https://www.amazon.com/gp/product/B0748MPQT4 https://www.amazon.com/gp/product/B07NY23WBG

running the june one-step install, updated. 80G cam 10M music (testing it out) AP SAMBA

[miles267]

@natrlhy that's almost exactly the RP0 setup I'm running with Buster image. It's the same RP0 kit, but I'm using a SanDisk Extreme Pro 128 GB microSD currently. 16G cam, 4G music. I misunderstood and thought the larger the cam partition the better. Rather, since setting it to a smaller, specific capacity seems to have improved reliability.

I couldn't even get the June Stretch one-step install to complete on the same kit. Tried repeatedly. Not sure why.

I've not enabled AP or SAMBA. What are the benefits?

[stepir]

So after one day of testing also my MX started reporting the USB storage to be too slow (less then 4MB/s mentioned the error). Restarting the PI solved the issue for the rest of the day but happpened again the day after...

[itsrainingben]

Almost a week now using the following config. No issues thus far and it's seen a decent amount of sentry events. The only difference I can tell/recall is the size allocation &, I have not run any updates on the card since imaging.

256GB Card: June raspbian preconfig image. 90% CAM 1GB Music No Updates

I had the following card ready to swap in but it's just been sitting in the console. Lonely. Waiting.

64GB Card: Septmber stretch preconfig image 90% CAM 1GB Music No Updates

Still would like to get to the bottom of this. Somewhat perplexing.

On Thu, Oct 10, 2019 at 1:57 PM stepir notifications@github.com wrote:

So after one day of testing also my MX started reporting the USB storage to be too slow (less then 4MB/s mentioned the error). Restarting the PI solved the issue for the rest of the day but happpened again the day after...

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/marcone/teslausb/issues/239?email_source=notifications&email_token=AGE3NGSTXW45IE555NTTLWTQN6JLTA5CNFSM4IUEPZCKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEA55GOQ#issuecomment-540791610, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGE3NGVUTDZNKZ4X6OUH7KDQN6JLTANCNFSM4IUEPZCA .

[Vipercat]

@infernix, any instructions available on how to re-partition the music partition to FAT32 instead of the current ext4?

[curiousgrge]

Update: So it's been 3 solid days and still no errors. I have tons of events and even captured events while driving. Still not one single error. My configuration is above.

added link for ease: #239 (comment)

Do you use a battery to keep the Pi0 from sleeping?

[wwwb0n3zcom]

Update: So it's been 3 solid days and still no errors. I have tons of events and even captured events while driving. Still not one single error. My configuration is above. added link for ease: #239 (comment)

Do you use a battery to keep the Pi0 from sleeping?

No I don't. But I have decent WiFi when I park in the driveway and it's archiving each time. So I don't get a large amount data to transfer. I'm archiving to the NAS via CIFS but have to set the following for that to work: export cifs_version="2.0"

Update - Just updated to 2019.32.12.1 after work. I removed the RPi0w before the update and inserted it after the Tesla firmware completed. The RPi0w is still working on this firmware too.

[dadealus]

24+ hours later and both are still working

Sandisk Extreme 64GB. Works no issue Samsung Evo Select 64GB Works no issue

Noticed both of these have the U3 certification as the Samsung Pro Endurance only has the U1 cert.

If you are having issue I would recommend you swap to the Sandisk Extreme Pro 64GB/128GB Leave at least 2GB free so ... go something like 90% cam size and music set to =128M

I'm not sure if setting music to 0 is supported but I would recommend that CamSize+MusicSize does not equal card size.. leave some room.

[stepir]

I've reimaged my PI, enabled CIFS which will make @marcone happy (kidding) and partition my SD (a kingston) for 85% CAM and 1GB MUSIC. This should be leaving enough space to better handle the flash housekeeping... for sure it's a write intensive workload and to a certain degree it could be the reason behind the issue

[stepir]

Update: So it's been 3 solid days and still no errors. I have tons of events and even captured events while driving. Still not one single error. My configuration is above. added link for ease: #239 (comment)

Do you use a battery to keep the Pi0 from sleeping?

No I don't. But I have decent WiFi when I park in the driveway and it's archiving each time. So I don't get a large amount data to transfer. I'm archiving to the NAS via CIFS but have to set the following for that to work: export cifs_version="2.0"

Update - Just updated to 2019.32.12.1 after work. I removed the RPi0w before the update and inserted it after the Tesla firmware completed. The RPi0w is still working on this firmware too.

why not v3? this is what I've enabled now (not sure what was the default before the explicit setting in config)

[ZeFrenchBibster]

H I've just re-flashed with the 20190615 image. Using NFS, so I had to edit /etc/fstab manually, and reinstall rpcbind (apt-get --reinstall install rpsbind). Om my Samsung EVO 128GB card, I did 90% CAM and 16Gb music.

I'll plug the rpi0W in the Model 3 later today and I'll report back.

[miles267]

I'm currently on a RP0 June Stretch install with CIFS 3.0 and a SanDisk Extreme 256GB microSD card. Camsize 100%, no music. So far, both dashcam and sentry mode have been working without issue. It's the dashcam icon that I tend to watch closely as it has been intermittent over the course of my v10 testing.

I kept my pre-configured SanDisk Extreme Pro 128GB card imaged with buster, just in case I need to pop that back into my RP0.

[dadealus]

H I've just re-flashed with the 20190615 image. Using NFS, so I had to edit /etc/fstab manually, and reinstall rpcbind (apt-get --reinstall install rpsbind). Om my Samsung EVO 128GB card, I did 90% CAM and 16Gb music.

I'll plug the rpi0W in the Model 3 later today and I'll report back.

90% of 128. Plus 16gb is 131Gb ...

[marcone]

90% of 128. Plus 16gb is 131Gb ...

Available space after OS install on a 128GB card is about 117 GB, and the setup script keeps 6GB in reserve, so 111GB available, effectively. The setup script then takes 90% of that, and music size is adjusted to fit, if needed. I think that if you run ls -lh /backingfiles/*.bin you'll see that it ended up with about 100GB for cam and about 11GB for music in this case. (I recommend just specifying both numbers as absolute values instead of percentages, and using a much smaller number for camsize, since the car won't effectively use that much storage anyway)

[retinoic]

Reflashed June release a couple days ago. Did not apply any updates and used my standard 75% cam config (on Samsung 128 GB high endurance). Sentry has been on & recorded a few events. No error so far, but I’m not holding my breath.

[dadealus]

90% of 128. Plus 16gb is 131Gb ...

Available space after OS install on a 128GB card is about 117 GB, and the setup script keeps 6GB in reserve, so 111GB available, effectively. The setup script then takes 90% of that, and music size is adjusted to fit, if needed. I think that if you run ls -lh /backingfiles/*.bin you'll see that it ended up with about 100GB for cam and about 11GB for music in this case. (I recommend just specifying both numbers as absolute values instead of percentages, and using a much smaller number for camsize, since the car won't effectively use that much storage anyway)

Yeah, after really understanding what's going on here under the hood. 64gb is the sweet spot. And most of these issues people are having is due to poor SD card performance. I've tested 8 different cards now and it seems U1 cards and cheap cards are the common issue here. As stated before if you have issues just return whatever you have pickup the 64GB SanDisk Extreme Pro and stop worrying about having enough space. The card is going to empty everytime you return within range to your SMB share.

Every U3 rated 64Gb card I have tested in any configuration works perfectly on Pi0 and Pi4. Also was tested In the S,X, and 3

[Vipercat]

Nothing but issues after the update to the Buster Image. USB too slow, etc. Tried all possible scenario's regarding cam size and music size, but all make the dashcam end up with a grey x at a certain point in time and the music content will not even load properly. Most music that is on the partition is not shown. Don't know why. I wish I could go back to not having the 'snapshot' function. I honestly do not understand why that is there anyway. Sorry! Having the SavedClips and in addition the SentryClips stored and moved to my NAS via cifs is all I even needed. I used to have 38GB for CAM and 12GB for music, 6GB reserved free space and all worked fine. Until the version with the snapshot functionality. Will analyze the logging and run a diagnostics and upload. I was so happy with all working well for months and now it broke. Too bad. Still very much appreciated al the work done @marcone, so don't get me wrong on this.

[Vipercat]

What I also noticed BTW is that now at V10 the 12V sigarette lighter port (model S) that I used to provide constant power to the RPI now loses power after x minutes when the car goes to sleep. As soon as I enable Sentry Mode the port is powered again. I used to use this port for power and the USB data port for data. I would like to keep the RPI powered on all the time, but enabling sentry while at home used to activate the alarm when the RPI started copying files and disconnecting USB for the car. Not sure if this is still the case, but I don't want to find out the hard way.

[th3fallen]

@marcone is there a way for us to disable snapshots if just to test with?

[dsgerbc]

@marcone is there a way for us to disable snapshots if just to test with?

Not him, but I've looked at this. No easy way to disable everything. So I forked the project and rolled it back to August 18th state. I tried just running the setup-script (with my repo/branch in the setup-teslausb.conf) from that state, it didn't give me errors, but I did get the 'USB too slow' this morning. I'm gonna do a fresh install off the June'19 image, and then update to the same pre-snapshot state and see if it helps. If it doesn't, I'm gonna keep rolling back commits until I get to July 11-ish - that's the version I had working without any major hitches (except backing up SentryClips to the archive, but that's an easy fix).

[th3fallen]

i've tried commenting out the lines that do the snapshots in the archive loop file. will report back with my findings

[dadealus]

@marcone is there a way for us to disable snapshots if just to test with?

Not him, but I've looked at this. No easy way to disable everything. So I forked the project and rolled it back to August 18th state. I tried just running the setup-script (with my repo/branch in the setup-teslausb.conf) from that state, it didn't give me errors, but I did get the 'USB too slow' this morning. I'm gonna do a fresh install off the June'19 image, and then update to the same pre-snapshot state and see if it helps. If it doesn't, I'm gonna keep rolling back commits until I get to July 11-ish - that's the version I had working without any major hitches (except backing up SentryClips to the archive, but that's an easy fix).

What SD card are you using?

[Jonnehs]

So, sorry, I've been away for a while. What is the fix? Reflash?

[CircuitGuy]

I'm having a bit of success now with my setup (48 hours) now: RPi Zero W Samsung PRO Endurance 128GB

Special thanks to the conversations here that mentioned the burst IO.

I made two changes:

1. Re-flashed to buster release and left some free space ~35% for filesystem management (60% CAM and 4 GB music). Did not solve the problem on its own.

2. Updated the Linux cache mechanism to background write at 2 kB and use 80% of RAM before foreground writing

Reference and my theory

https://lonesysadmin.net/2013/12/22/better-linux-disk-caching-performance-vm-dirty_ratio/

We really want the raspberry Pi to cache everything it can in RAM and slowly offload to the SD card. Note - I have a battery in-line so I'm not worried about a power disconnect data loss. I suspect that a foreground priority write to the SD card may lock the CPU enough that Tesla sees the hiccup and rejects the storage device.

Defaults on the raspberry pi:

vm.dirty_background_ratio = 10
vm.dirty_ratio = 20

I changed these through editing /etc/sysctl.conf and running sysctl –p to:

vm.dirty_background_bytes = 2048
vm.dirty_ratio = 80

My theory is that we want the RPi to immediately offload any waiting IO via a low-priority background process. I.e. only waiting for 2 kB to begin writing. Similarly, we want to buffer everything we possibly can (80% of RAM use) before forcing a non-background write.

The RPi has 512 MB of RAM and the OS uses ~40 in my case (= 472 MB free). The Tesla will burst write ~110 MB (all 4 cameras) at a time. With the defaults (20% / 472 MB RAM) on a RPi a data burst of only 95 MB would guarantee a priority foreground write, potentially stalling the CPU.

This also potentially explains why v10 broke this in combination with updating unrelated OS things. V10 added the extra camera, bringing the burst from ~80MB to 110MB and random unrelated OS things could eat into free RAM, pushing us over that precarious 20% background write buffer. A faster card could mitigate this, locking the RPi CPU for less time during the foreground write.

[miles267]

@circuitguy what were your exact sizes you used on that microSD card for both Camsize and music? I may try to match your setup exactly. Also are you using buster or stretch image?