search

marcone / teslausb

A smart USB drive for Tesla Dashcam - extended storage, auto archive, web viewer
MIT License
1.85k stars 337 forks source link

USB Storage Slow? #239

Open DonC52 opened 4 years ago

DonC52 commented 4 years ago

I have been wondering why the following occurs on my Windows 10 laptop and the Pi USB. I am using a 128G SanDisk Extreme Plus UHS so I would think it rather fast.

I put in an old "slow" USB stick into my Laptop port and copy a Sentry Mode directory, I get about 20MB/Sec transfer rate.

I put the Pi USB in with power applied from external source and copy the same directory, I get like 1.5MB/s to 2.5MB/s transfer rate.

Looking at top, there is a LOT of wait time going on. Is there anything that can be done to speed things up?

top

dadealus commented 4 years ago

So I've talked with some people and they are running pi0 and not seeing the issue. They are also running v10 on M3. Hmm

miles267 commented 4 years ago

@dadealus who are these people? And what configuration are they running? Seems unlikely but we’re all hopeful.

vita10gy commented 4 years ago

@dadealus who are these people? And what configuration are they running? Seems unlikely but we’re all hopeful.

I think @marcone for one, which kind of bites. The person who might most understand what to do about it isn't seeing the issue to debug it.

dadealus commented 4 years ago

Chris from Crosstalk and my buddy. Both are using the parts in his guide. June image, 80% cam size, and no tweaks.

marcone commented 4 years ago

The only thing I've found so far that improves speed is to overclock the sd card. A high quality card should be able to support that, but you may also render the Pi unbootable. If you want to try it, add the following to /boot/config.txt:

dtparam=sd_overclock=100

and then reboot. If it doesn't work for you, you can try lower values until it works. Default value is 50, so no point in going lower than that.

vita10gy commented 4 years ago

This might be a dumb question, but if we do that and it doesn't boot does that mean the only way to try a new value is by reflashing the whole process over again?

marcone commented 4 years ago

If it simply doesn't boot, you can put the sd card in a PC and remove that line from config.txt. The only reason to reflash would be if it ended up corrupting the card.

vita10gy commented 4 years ago

oh, doy. I was trying to remember where all that was and completely forgot about putting the card in directly.

It booted with overclock set to 100 (though I'm not sure if there's a way to confirm it worked) but no luck on fixing the issue there.

dadealus commented 4 years ago

so far setting fresh june image with 80% camsize. no error. will report back after another 24 hours.

curiousgrge commented 4 years ago

Reimaged last night using latest buster beta release 20190922 after having USB drive write speed too slow. On latest 2019.32.11.1 firmware applied two nights ago. Applied updating instructions and ran teslausb-setup again. Applied nofua=1,1 this time and didn't notice a change since the last time I copied my music files over. Took about 32 minutes compared to what I reported the last time which was about 30 minutes. I have since added another couple hundred files and I think mostly this time it was MP3 rather than FLAC so files are much smaller. So I think modifying the Windows disk device driver makes a bigger impact than nofua=1,1 with it still hoving around 15-20MB/sec during the copy process. Didn't have a problem this morning with the error message but the day isn't done yet.

@curiousgrge Is this still working for you? And how do you add this nofua=1,1 setting to an existing setup?

It appears to still be working me but the message has come up at least once while sitting in a parking lot for a whole day with sentry mode on. Since I have excluded home from sentry events, I haven't had the message pop up for shorter drives.

nofua=1,1 was explained where it needs to go from marcone. But if you're having problems saving the change, then it would go something like this from ssh:

sudo -i ./root/bin/remountfs_rw nano /etc/modprobe.d/g_mass_storage.conf

then save, exit. reboot
marcone commented 4 years ago

It booted with overclock set to 100 (though I'm not sure if there's a way to confirm it worked) but no luck on fixing the issue there.

So just to confirm: you set overclock to 100, and still had the "usb too slow" error after that?

vita10gy commented 4 years ago

It booted with overclock set to 100 (though I'm not sure if there's a way to confirm it worked) but no luck on fixing the issue there.

So just to confirm: you set overclock to 100, and still had the "usb too slow" error after that?

Yes, however after a reboot I got an error again and the error was actually about it not being formatted. So, it's possible it didn't boot reliably with that setting and I just assumed the error the first time. Trying again at 80.

vita10gy commented 4 years ago

Yeah, it just complains the drive isn't accessible and it needs to be reformatted. The USB music shows up and plays fine though, so I don't know what gives. I did add the nofua=1,1, after just making it overclocked to 100 didn't work, but presumably if that screwed everything up the music wouldn't play.

I tried the self updating process and that didn't seem to force a reformatting of whatever the car thinks the issue is either. Windows sees the cam folders fine when you plug it in.

miles267 commented 4 years ago

@dadealus is your fix still working after 24 hours? If so, I’d consider it a success.

dadealus commented 4 years ago

@miles267 So I actually got a Pi4 delivered and noticed home was stopping my sentry mode. so not sure... so What I did was redid the Pi0 on my 3 and enabled sentry on all locations. and its outside my garage and will get quite a bit of random activations and short drives. so it should trigger easy over the next few days. On my X I'm running the latest pi4 and image with updates. not running any tweaks same settings and I'll be parking it downtown tomorrow so should be PLENTY of activations. After Im going to swap them and see if its car hardware effected. but so far it seems the M3 with the pi0 is working without any issues.

ClanLee commented 4 years ago

On V9, no issues. After upgrading to V10, I was getting the same issue, USB too slow after a few hours. I was reading the optional setup information about allocating SD Storage. It's recommended (but doesn't explain why) to leave unallocated space on the SD card. Initially, I utilized the entire 128 GB card by assigning 65% to Cam and 35% to Music but started to received USB too slow.

I played with a few settings and finally settled on 65 GB for Cam and 15 GB for music, leaving 38 GB of unallocated space (118 GB - 65 GB - 15 GB = 38 GB). I've had this running for over 12 hours and everything is working correctly. I captured sentry events and saved events and the Pi 0 is working without any issues.

@marcone Can you explain why you recommend leaving unallocated space on the SD card?

curiousgrge commented 4 years ago

So I've had the message up crop up at least 3x this weekend with my limited driving. Unplugging and plugging back in appears to reset it but it's only a matter of time before this will happen again.

dadealus commented 4 years ago

@miles267 Walked outside to the Pi4 (latest image and patch) in the Raven X AP3. USB too slow message. So Im even seeing this issue on the Pi4. went ahead and swapped USB ports on the X to see if that changes anything. SD card sustains writes in the 80+MB/s when directly inserted to PC .. Model 3 with AP3 on the other hand is running june image without any updates (cam size set to 80%) and its still going no issues. both cars are on V10. Will continue to report.

miles267 commented 4 years ago

I'm now 2 days into using my Pi4 buster image (latest) and all available updates in M3 AP3 without issue. It's also being powered off a USB-A to USB-C cable only. If it matters, it's a 128 GB SanDisk Ultra microSD card with camsize=32G.

That said, I'm still a bit confused why I'd not set the camsize=100% of the available card space instead of 25% (32G of 128G available) when I'm not using the remaining space for music or anytihng else.

vita10gy commented 4 years ago

I reflashed the june build and it seems ok now. I've barely left the garage, but the car/drive bombed out in the garage a few times before.

The only settings differences (IIRC) is that I specified 10% for the music to leave 20% of the (64gb) card for teslausb magic, and I didn't enable the samba server.

marcone commented 4 years ago

That said, I'm still a bit confused why I'd not set the camsize=100% of the available card space instead of 25% (32G of 128G available) when I'm not using the remaining space for music or anytihng else.

Because that 65GB of "CAM" drive will only be used if you accumulate 8 hours worth of Sentry events between archives. Otherwise it's just wasted space that teslausb could be using to save more than 1 hour of RecentClips.

vita10gy commented 4 years ago

Tesla is so close to the ideal set up now. It relied on separating saved and sentry, which they did, and it's a shame they didn't go the rest of the way. What it SHOULD do is.

Saved Clips - Stay Forever until drive is full. You explicitly saved these. Presumably there's a reason. Sentry Clips - Fill the remaining drive. If needed remove, in order, the oldest recent clips, then the oldest sentry clips. Recent Clips - Fill the remaining drive, remove oldest Recent as needed.

If you throw petabyte at the Teslacam it should fill it with something.

DineshCyanam commented 4 years ago

I have reflashed the June image with 100% CAM size (Samsung 128GB PRO Endurance SD Card) and on my 25 mile drive to work, everything seemed OK with no errors. The car is now parked and charging with Sentry turned on. Will report back this evening to see if the Pi Zero is still working.

natrlhy commented 4 years ago

I also did a reflash with a new 128GB card, but sitting in a parking lot I got the slow error. I also have export camsize=100% I'm wondering if I set it to say, 100GB (I don't need music so the rest would be free space) if that would help. Hoping this is the solution as it seems we are honing in on a workable setup with no errors on the Rpi0's perhaps...

itsrainingben commented 4 years ago

Give it a try and report back! The more data points and information we have the better.

This evening I'll be flashing two cards with the following differences, both cards are the same make and will be allocating roughly 90% of storage to the CAM partition. No Music will be loaded or played.

256GB - Buster image, no updates 64GB - June image, no updates

I'll roll with both throughout the week and report back.

On Mon, Oct 7, 2019 at 9:52 AM natrlhy notifications@github.com wrote:

I also did a reflash with a new 128GB card, but sitting in a parking lot I got the slow error. I also have export camsize=100% I'm wondering if I set it to say, 100GB (I don't need music so the rest would be free space) if that would help. Hoping this is the solution as it seems we are honing in on a workable setup with no errors on the Rpi0's perhaps...

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/marcone/teslausb/issues/239?email_source=notifications&email_token=AGE3NGXKAFP3N5655LPIDODQNNSLTA5CNFSM4IUEPZCKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEARBOMA#issuecomment-539105072, or mute the thread https://github.com/notifications/unsubscribe-auth/AGE3NGVSMWNQ7KIKR7EJMHTQNNSLTANCNFSM4IUEPZCA .

ClanLee commented 4 years ago

That said, I'm still a bit confused why I'd not set the camsize=100% of the available card space instead of 25% (32G of 128G available) when I'm not using the remaining space for music or anytihng else.

Because that 65GB of "CAM" drive will only be used if you accumulate 8 hours worth of Sentry events between archives. Otherwise it's just wasted space that teslausb could be using to save more than 1 hour of RecentClips.

This is what is unclear to me. If we allocate 65 GB for "CAM", wouldn't that be used for all Recent, Saved and Sentry Clips? If that isn't the case, the setup information should specify to leave unallocated space for Recent and Saved Clips. I know that I wanted to maximize the use of the card for TeslaCam.

marcone commented 4 years ago

This is what is unclear to me. If we allocate 65 GB for "CAM", wouldn't that be used for all Recent, Saved and Sentry Clips?

If you allocate 65GB for "CAM", then the car will see a 65GB drive. The car only keeps about 7.2 GB of RecentClips though (because that's how much it records per hour, and deletes RecentClips that are older than one hour), so that leaves 8 hours worth of Saved/Sentry clips. Since those are deleted from the drive when teslausb archives them, having space for 8 hours of Saved/Sentry clips is only useful if you actually get that much footage between archiving.

dadealus commented 4 years ago

Right, so maybe the PI is having issues with these larger cards / partitions ? alot of people here are using 128gb+

dadealus commented 4 years ago

Update on the testing ... my Pi4 with latest image and updates with 128gb card crapped out 3 times today over 3 hours. was on a MXr /w AP3

M3 SR+ w/ AP3 Pi0 june image 0 updates still going

wwwb0n3zcom commented 4 years ago

So just to add another data point.

Make: Model 3 Performance Delivery: July 28th 2018 Hardware: 2.5 Firmware: 2019.32.11.1 Device: Raspberry Pi Zero W Memory: SanDisk High Endurance 256GB

I left the nofua=1,1 within the _g_massstorage.conf. But that didn't seem to help. So on Sunday I unmounted and removed the backingfiles and snapshots. I then reconfigured the _teslausb_setupvariables.conf with export camsize=128G and export musicsize=80G. Today (Monday) is the first day I have gone an entire day without an error on the screen about a slow drive. I wonder if the extra 48G (minus the space needed by the OS) is helping this...

EvolvingSoftware commented 4 years ago

Has it happened to anyone who’s not using the MP3 functionality?

Might be unrelated but I was having issues previously when using a USB drive and doing the split partitions to have the TeslaCam and MP3s on the same stick. I wonder if the tolerance is less when it’s trying to use it for both activities?

ice2032 commented 4 years ago

I’m not using the music functionality at all and it still craps out. This is basically useless for me until we figure out the issue. Can’t have a dash cam that only records for 30 mins to an hour and then never works again.

On Oct 7, 2019, at 7:59 PM, roisterous notifications@github.com wrote:

 Has it happened to anyone who’s not using the MP3 functionality?

Might be unrelated but I was having issues previously when using a USB drive and doing the split partitions to have the TeslaCam and MP3s on the same stick. I wonder if the tolerance is less when it’s trying to use it for both activities?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

DineshCyanam commented 4 years ago

I have reflashed the June image with 100% CAM size (Samsung 128GB PRO Endurance SD Card) and on my 25 mile drive to work, everything seemed OK with no errors. The car is now parked and charging with Sentry turned on. Will report back this evening to see if the Pi Zero is still working.

Update: Pi Zero has worked throughout the day (~10 hours) with no errors.

tkunstek commented 4 years ago

Just wanted to give some quick feedback in case it helps others.

The Problem

I was getting repeated USB drive to slow messages from my Tesla Model S while just setting in my garage with sentry mode off and dash cam running.

My Setup

Settings

Video at 40% usage Music at 20% usage Sync using rsync, but using an alternate port number so I had to modify one line in one file, otherwise stock image.

Resolution

I was using a crappy 3 ft USB 1 cable, one of the twenty you have a box of. Paper thin but otherwise never used. I switched to a thick, USB 2.1 cable (linked above) and it solved every single problem. Really don't discount the quality of the USB cable folks.

CircuitGuy commented 4 years ago

Want to throw in that I realized after v10, data wasn't getting copied for about a week. Ran the selfupdate procedure and found the "USB drive write speed is too slow for Dashcam" error occasionally. Re-plugging the RPi Zero W fixes the issue. The issue seems to appear for me when driving, not when I get in the car after having sentry mode on.

RPi Zero W Samsung PRO Endurance 128GB

So: I'm one of the people running the old OS. I had known good operation before the self-update.

I'm using a very short (several inch) USB cable, in-line battery, AND the whole setup is inside a mount so it doesn't move. The update was done over SSH - so I can also eliminate mechanical jiggling.

First debug step

Edit: Did not work

I tried upping my SD card frequency to 75 MHz. Nearest valid is 66.6 MHz: dtparam=sd_overclock=75 

pi@teslausb:~ $ sudo grep clock /sys/kernel/debug/mmc0/ios
clock:          50000000 Hz
actual clock:   66666666 Hz
vita10gy commented 4 years ago

I switched to a thick, USB 2.1 cable (linked above) and it solved every single problem. Really don't discount the quality of the USB cable folks.

I switched to that cables/brands right angled brother and it didn't fix it.

wwwb0n3zcom commented 4 years ago

I know it's not a cable issue in my scenario. This is the same cable I have used before v10 and it's from a reputable manufacture: https://www.amazon.com/gp/product/B01NAMTC5T

I'll keep everyone posted as days go by if it fails.

itsrainingben commented 4 years ago

If the cable works initially, that should not be the variable in this instance.

On Mon, Oct 7, 2019 at 9:33 PM Kenneth Jones notifications@github.com wrote:

I know it's not a cable issue in my scenario. This is the same cable I have used before v10 and it's from a reputable manufacture: https://www.amazon.com/gp/product/B01NAMTC5T

I'll keep everyone posted as days go by if it fails.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/marcone/teslausb/issues/239?email_source=notifications&email_token=AGE3NGUBDAZMMKMVHC3FCCTQNQESBA5CNFSM4IUEPZCKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEASVDOY#issuecomment-539316667, or mute the thread https://github.com/notifications/unsubscribe-auth/AGE3NGTMCUTQGSDVOGUWGW3QNQESBANCNFSM4IUEPZCA .

githubsean commented 4 years ago

I keep thinking about whether it's the fact that the USB drive presented to the Tesla (or a computer) is from the g_mass_storage driver, and then the backing store for this is not real partition, but just a file. Then this gets formatted as fat32.

Now I believe that the fat32 driver is aware of the underlying hardware - if it was directly pointing to a device partition - i.e. /dev/sdc1 (or something similar) where /dev/sdc is the USB device. Since we have a partition on a (fake) device which is backed by a file on a ext4 partition on a real device, somewhere along the line is going to be lost any information about the alignment of blocks of flash memory to file system blocks. Any optimisations and behaviours that the fat32 driver could perform if it was writing direct to the flash device can't be done. Also, there is no guarantee that the file backing the storage is aligned to the underlying USB storage device, or that the blocks are contiguous. This will means that there might be significant write amplification (single writes leading to multiple writes across several blocks) when writing to the fat32 drive.

Is there any possibility of instead directly partitioning the USB drive so that that the partition start can be aligned with the underlying physical storage (perhaps on a 4k boundary), and that there is a contiguous sequence of blocks of physical storage allocated to the fat32 drive. For example, if you had a 64GB SD card, you could reserve a few GB for the Pi boot partition, and the main filesystem partition (the 1st two partitions), and then create a third that takes up the rest of the space.

Or, is there a way to have the backing storage for the fat32 partition aligned and contiguous?

These are just my musings - I'm just thinking that if we can reduce the number of layers of software between a write to the TeslaCam drive and a write to the underlying USB storage would be a "good thing" (tm)

edalquist commented 4 years ago

I'm wondering the same thing, further the folks that have better luck with some slack space might be further evidence. Some discussion about just ext4, sd cards and wear leveling: https://www.raspberrypi.org/forums/viewtopic.php?t=19554

I just re-flashed to the june image with the config pointing at a fork with a branch hung off the june release: https://github.com/edalquist/teslausb/tree/june-release

Could someone who is still seeing the issue try running fstrim -v / on their pi? That should result in sending TRIM commands to microsd controller addressing potential fragmentation issues. I sort of doubt that is the issue but it is worth a test.

shawnbissell commented 4 years ago

So just to add another data point.

Make: Model 3 Performance Delivery: July 28th 2018 Hardware: 2.5 Firmware: 2019.32.11.1 Device: Raspberry Pi Zero W Memory: SanDisk High Endurance 256GB

I left the nofua=1,1 within the _g_massstorage.conf. But that didn't seem to help. So on Sunday I unmounted and removed the backingfiles and snapshots. I then reconfigured the _teslausb_setupvariables.conf with export camsize=128G and export musicsize=80G. Today (Monday) is the first day I have gone an entire day without an error on the screen about a slow drive. I wonder if the extra 48G (minus the space needed by the OS) is helping this...

I've just starting testing this exact same setup except using the Buster version with a SanDisk High Endurance 128GB with about 18GB free space. I've also turned Sentry Mode to always on (even at home) to try and provoke the problem.

miles267 commented 4 years ago

Unfortunately one day isn’t enough time. If you go 3 days without the error, it could be success.

infernix commented 4 years ago

So just to confirm: you set overclock to 100, and still had the "usb too slow" error after that?

Yes for me. Rpi 0W with samsung pro 64gb endurance. So this does not seem enough to fix it.

itsrainingben commented 4 years ago

I think this is valuable insight Sean. Curious what @marcone/teslausb teslausb@noreply.github.com thinks about presenting the partitions as actual/direct mount points.

I'm still running my tests as outlined above. Nothing to report at this juncture, too early.

On Mon, Oct 7, 2019 at 11:25 PM Sean Anderson notifications@github.com wrote:

I keep thinking about whether it's the fact that the USB drive presented to the Tesla (or a computer) is from the g_mass_storage driver, and then the backing store for this is not real partition, but just a file. Then this gets formatted as fat32.

Now I believe that the fat32 driver is aware of the underlying hardware - if it was directly pointing to a device partition - i.e. /dev/sdc1 (or something similar) where /dev/sdc is the USB device. Since we have a partition on a (fake) device which is backed by a file on a ext4 partition on a real device, somewhere along the line is going to be lost any information about the alignment of blocks of flash memory to file system blocks. Any optimisations and behaviours that the fat32 driver could perform if it was writing direct to the flash device can't be done. Also, there is no guarantee that the file backing the storage is aligned to the underlying USB storage device, or that the blocks are contiguous. This will means that there might be significant write amplification (single writes leading to multiple writes across several blocks) when writing to the fat32 drive.

Is there any possibility of instead directly partitioning the USB drive so that that the partition start can be aligned with the underlying physical storage (perhaps on a 4k boundary), and that there is a contiguous sequence of blocks of physical storage allocated to the fat32 drive. For example, if you had a 64GB SD card, you could reserve a few GB for the Pi boot partition, and the main filesystem partition (the 1st two partitions), and then create a third that takes up the rest of the space.

Or, is there a way to have the backing storage for the fat32 partition aligned and contiguous?

These are just my musings - I'm just thinking that if we can reduce the number of layers of software between a write to the TeslaCam drive and a write to the underlying USB storage would be a "good thing" (tm)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/marcone/teslausb/issues/239?email_source=notifications&email_token=AGE3NGQLUMAX2BBXI3KA4X3QNQRW7A5CNFSM4IUEPZCKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAS7QUQ#issuecomment-539359314, or mute the thread https://github.com/notifications/unsubscribe-auth/AGE3NGUFFKDYSMV23CLDBXDQNQRW7ANCNFSM4IUEPZCA .

marcone commented 4 years ago

I think this is valuable insight Sean. Curious what @marcone/teslausb teslausb@noreply.github.com thinks about presenting the partitions as actual/direct mount points.

That's a nonstarter for me, as it would prevent snapshots from working, which in turn prevents extended recentclips and samba access from working.

(I also doubt that alignment is the issue here, given that people are reporting "usb too slow" using usb drives directly without teslausb)

dadealus commented 4 years ago

So I am trying out various SD cards. I have a few theories.

CircuitGuy commented 4 years ago

@dadealus Your results don't discount this theory:

https://forums.tesla.com/forum/forums/dashcam-usb-drive-too-slow-save?page=2

Some Tesla forum people seem to think that cards > 32 GB are causing an issue.

On this thread, people have found that lowering the camera partition size helped. I wonder if they're splitting it below 32 GB?

I'll try partitioning to 32 GB to see if that solves the issue. If so, it means TeslaCam could fix it and keep more storage space by "hiding" the extra partition.

Edit: The speed requirement in the Tesla error message is only 4 MBps. Almost any card should be able to hit this rate.

miles267 commented 4 years ago

I've already tried to set my camsize=16G or 32G (althought it's 128 GB) without success.

dadealus commented 4 years ago

I've already tried to set my camsize=16G or 32G (althought it's 128 GB) without success.

@dadealus Your results don't discount this theory:

https://forums.tesla.com/forum/forums/dashcam-usb-drive-too-slow-save?page=2

Some Tesla forum people seem to think that cards > 32 GB are causing an issue.

On this thread, people have found that lowering the camera partition size helped. I wonder if they're splitting it below 32 GB?

I'll try partitioning to 32 GB to see if that solves the issue. If so, it means TeslaCam could fix it and keep more storage space by "hiding" the extra partition.

Edit: The speed requirement in the Tesla error message is only 4 MBps. Almost any card should be able to hit this rate.

So I think it might be a mixture of the two. My Samsung Pro Endurance no matter what (32gb cam size or 100gb bombs out). I think some of these cards just cannot handle sustained long term writes based on how they are built.

my next steps are to try the 64gb cards from SanDisk Extreme and Samsung Evo Select. going to try 100% camsize and then something like 80% camsize, 2GB music, and the remaining left for the OS.

c-E-c-777 commented 4 years ago

FWIW I'm using a SanDisk Extreme 128GB, and haven't had any issues in the last week since switching over to the Buster image. I've set my cam partition to 64GB and music to 8GB