Imager goes above 100%

[Bombardier-C-Kram]

Im not really sure what I did... I did cancel some OS installs a few times and now it goes a lot above 100%. Is this a known bug?

[maxnet]

Linux platform?

[Bombardier-C-Kram]

Yeah, Ubuntu 20.04

On Mon, 22 Jun 2020 at 16:41, maxnet notifications@github.com wrote:

Linux platform?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/raspberrypi/rpi-imager/issues/82#issuecomment-647732350, or unsubscribe https://github.com/notifications/unsubscribe-auth/AMIHRRZR6XCMSCZPIYIB3TTRX6XVJANCNFSM4OE6Q6KQ .

--

SpaceQuacker | General Support L2

| FreeMcServer.net

| spacequacker@freemcserver.net

| freemcserver.net https://freemcserver.net/discord https://twitter.com/spacequacker https://www.facebook.com/freemcservernet https://github.com/Bombardier-C-Kram https://gitlab.com/Bombardier-C-Kram

[Bombardier-C-Kram]

I also noticed that if i tried to use etcher, I would get really slow speads... Like speads of just 90mb per second or something... Like 1 hour to do one flash of raspberry pi OS

[maxnet]

Probably still writing out the previous write sessions you cancelled in the background, and that does not go well with the write progress code that asks kernel how much has been written to the SD card block device.

"Cancel" is not really effective on Linux, as it has huge write buffers. Once Imager has copied everything that needs to be written to the kernel's buffers -which can occur within seconds if you are writing a local file- there is no effective way for us to cancel the operation.

[Bombardier-C-Kram]

Ah okay. I just did it on a windows partition and it all worked well. Thanks

On Tue, 23 Jun 2020 at 08:12, maxnet notifications@github.com wrote:

Probably still writing out the previous write sessions you cancelled in the background, and that does not go well with the write progress code that asks kernel how much has been written to the SD card block device.

"Cancel" is not really effective on Linux, as it has huge write buffers. Once Imager has copied everything that needs to be written to the kernel's buffers -which can occur within seconds if you are writing a local file- there is no effective way for us to cancel the operation.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/raspberrypi/rpi-imager/issues/82#issuecomment-648078005, or unsubscribe https://github.com/notifications/unsubscribe-auth/AMIHRR7EXNGV467IKSBXVWLRYCE3DANCNFSM4OE6Q6KQ .

--

SpaceQuacker | General Support L2

| FreeMcServer.net

| spacequacker@freemcserver.net

| freemcserver.net https://freemcserver.net/discord https://twitter.com/spacequacker https://www.facebook.com/freemcservernet https://github.com/Bombardier-C-Kram https://gitlab.com/Bombardier-C-Kram

[lurch]

Once Imager has copied everything that needs to be written to the kernel's buffers -which can occur within seconds if you are writing a local file- there is no effective way for us to cancel the operation.

Would using O_DIRECT help with that? :shrug:

[maxnet]

Would using O_DIRECT help with that?

Yes, but then we wouldn't be able to take advantage of the kernel's highly efficient code to write from buffer to device.

Using O_DIRECT by-passes the kernel in talking to devices, and using it is pretending we know better how to pass data to devices than the kernel does. Ask the average kernel developer what they think about that :-)

http://lwn.net/2002/0516/a/lt-deranged-monkey.php3

The thing that has always disturbed me about O_DIRECT is that the whole interface is just stupid, and was probably designed by a deranged monkey on some serious mind-controlling substances

It's simply not very pretty, and it doesn't perform very well either because of the bad interfaces (where synchronocity of read/write is part of it, but the inherent page-table-walking is another issue).

(Quote from Linus T.)

[lurch]

I assumed for large-block-writes that it'd be beneficial to bypass the disk caches (see e.g. https://github.com/balena-io/etcher/issues/1523 ), but it sounds like you understand this much better than I do :slightly_smiling_face: (it was ages ago that I looked at all this, so perhaps I'm getting my flag names / nomenclature muddled up)

[Bombardier-C-Kram]

Ok so its a linux problem?

[maxnet]

I assumed for large-block-writes that it'd be beneficial to bypass the disk caches

Maybe if one has code that is more optimized for that purpose than what we have now. But right now Imager is much faster under Linux than under Windows and Mac OS X that have practically no write caching.

I assumed for large-block-writes that it'd be beneficial to bypass the disk caches (see e.g. balena-io/etcher#1523 )

We do currently use O_DIRECT for reads (enable it through fcntl after the write) to prevent the read from cache problem. Although it would probably be better to see if there is another way to drop cache (perhaps through posix_fadvise?) as well, as that is far from ideal, as it also disables read-ahead caching. (Dropping block cache through /proc/sys/vm is not possible, as we are not running as root)

Ok so its a linux problem?

That progressbar does not get over 100% is fixable. But accurate progress indication is a challenge under Linux, yes.

[lurch]

Yeah, I figured you'd probably have a much better understanding of this than my vague recollections :laughing:

Probably still writing out the previous write sessions you cancelled in the background, and that does not go well with the write progress code that asks kernel how much has been written to the SD card block device.

Hmm, so just out of curiosity: does that mean if you start a write, cancel it and then start a second write, Linux is actually still writing out the first image onto the SD card shortly before overwriting it with the second image?

[maxnet]

Hmm, so just out of curiosity: does that mean if you start a write, cancel it and then start a second write, Linux is actually still writing out the first image onto the SD card shortly before overwriting it with the second image?

On my own Kubuntu 18.04 Linux box pressing "cancel" in Imager will result in the old image being written to disk first -to the extend it was already written to disk cache-, before it even returns to the user interface, and allows you to select a new image. It seems to implicit flush everything to disk upon us closing the file descriptor with close().

If an implicit flush on close does not happen with the kernel version the TS is using, I would expect whether it writes the old image to SD card first to depend on how fast the new write occurs. If the new write is about writing a local image file to SD card and the TS has a lot of memory, I would expect the entire block cache to be replaced very fast. Once information in blocks of the cache is replaced, Linux will no longer write the older information of those blocks to disk. However if things are not so fast, for example because it has to download the new image from Internet, it may indeed continue to write some blocks from the old image in the background, yes. Same if there is not so much memory. If the size of write cache gets too large (a limit which is based on total memory size), it blocks new writes until everything in cache is flushed to disk first.

[lurch]

I wonder whether delaying the start-of-writing by a small amount, and using the delay to see if "something" is still being written to the SD card, would allow you to determine whether an old write-in-progress is still occurring? And thus you could "wait until the SD card settles" before starting the new write, which would enable you to calculate more accurate progress statistics? Hmmm, but then again if a multi-gigabyte image is already in the write-cache, I guess you probably want to get the new image into the write-cache as soon as possible, such that it "invalidates" the old data. Hmmm number 2: If you started flashing an 8GB image to an SD card (and assuming your Linux box has LOADS of free memory), and the whole 8GB got loaded into the write-cache, and then you cancelled the 8GB flash (with it continuing in the background), and you then selected a different 4GB image to flash, would Linux then load the 4GB image into the write-cache but end up writing the first 4GB of the second image onto the SD card, followed by the second half of the 8GB image? IYSWIM :wink:

[maxnet]

Hmmm number 2: If you started flashing an 8GB image to an SD card (and assuming your Linux box has LOADS of free memory)

By default Linux will block on executing write(), and wait before continuing until everything is flushed, once dirty page buffers ("write cache") has grown beyond 20% of total memory (changeable through /proc/sys/vm/dirty_ratio ) So only if the system had 40 GB of memory, would there be a chance the whole 8 GB would be in memory.

Hmm, thinking about that. The 1282% while writing Raspberry Pi OS (which is quite some GB) the TS gets should then not be possible to be caused by this particular problem. Might be something else. I do see an internal SD card reader is used to write, perhaps that device is reporting statistics in a different way than Imager is expecting. Not sure what exactly that could be though (different block size? perhaps the TRIM/DISCARD commands we sent result incorrect numbers? or something else, but hard to debug not having the same hardware).

[Bombardier-C-Kram]

The SDCard reader is the one from the pi desktop kit... Also, It would kinda hang and not even respond if I tried to kill it. As soon as I took it out of the sd card slot, it killed.

On Tue, 23 Jun 2020 at 16:47, maxnet notifications@github.com wrote:

Hmmm number 2: If you started flashing an 8GB image to an SD card (and assuming your Linux box has LOADS of free memory)

By default Linux will block on executing write(), and wait before continuing until everything is flushed, once dirty page buffers ("write cache") has grown beyond 20% of total memory (changeable through /proc/sys/vm/dirty_ratio ) So only if the system had 40 GB of memory, would there be a chance the whole 8 GB would be in memory.

Hmm, thinking about that. The 1282% while writing Raspberry Pi OS (which is quite some GB) the TS gets should then not be possible to be caused by this particular problem. Might be something else. I do see an internal SD card reader is used to write, perhaps that device is reporting statistics in a different way then Imager is expecting. Not sure what exactly that could be though (different block size? perhaps the TRIM/DISCARD commands we sent result incorrect numbers? or something else, but hard to debug not having the same hardware).

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/raspberrypi/rpi-imager/issues/82#issuecomment-648380520, or unsubscribe https://github.com/notifications/unsubscribe-auth/AMIHRR2NFHJRDZXACMIEQVDRYEBDDANCNFSM4OE6Q6KQ .

--

SpaceQuacker | General Support L2

| FreeMcServer.net

| spacequacker@freemcserver.net

| freemcserver.net https://freemcserver.net/discord https://twitter.com/spacequacker https://www.facebook.com/freemcservernet https://github.com/Bombardier-C-Kram https://gitlab.com/Bombardier-C-Kram

[Bombardier-C-Kram]

No, Like an sd card with a MicroSD in the back. The one in the official desktop kit.

On Tue, 23 Jun 2020 at 16:57, maxnet notifications@github.com wrote:

The SDCard reader is the one from the pi desktop kit...

So a USB one? Was a bit confused by "INTERNAL SD CARD" in your screenshot

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/raspberrypi/rpi-imager/issues/82#issuecomment-648385291, or unsubscribe https://github.com/notifications/unsubscribe-auth/AMIHRR272J3VXYCZZKAZ4NDRYECJRANCNFSM4OE6Q6KQ .

--

SpaceQuacker | General Support L2

| FreeMcServer.net

| spacequacker@freemcserver.net

| freemcserver.net https://freemcserver.net/discord https://twitter.com/spacequacker https://www.facebook.com/freemcservernet https://github.com/Bombardier-C-Kram https://gitlab.com/Bombardier-C-Kram

[lurch]

No, Like an sd card with a MicroSD in the back.

Ahhh, that's just an adaptor, used to convert the pins from a microSD card to a full-size SD card. The "reader" @maxnet is asking about is how is your SD-card-in-the-adaptor connected to your computer?

[Bombardier-C-Kram]

Its connected in an sd card slot. I use a 2011 macbook pro which has an internal sd card slot.

On Tue, 23 Jun 2020 at 17:05, Andrew Scheller notifications@github.com wrote:

No, Like an sd card with a MicroSD in the back.

Ahhh, that's just an adaptor, used to convert the pins from a microSD card to a full-size SD card. The "reader" @maxnet https://github.com/maxnet is asking about is how is your SD-card-in-the-adaptor connected to your computer?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/raspberrypi/rpi-imager/issues/82#issuecomment-648389116, or unsubscribe https://github.com/notifications/unsubscribe-auth/AMIHRR45FQ3J3J3OH7Z6AUDRYEDHXANCNFSM4OE6Q6KQ .

--

SpaceQuacker | General Support L2

| FreeMcServer.net

| spacequacker@freemcserver.net

| freemcserver.net https://freemcserver.net/discord https://twitter.com/spacequacker https://www.facebook.com/freemcservernet https://github.com/Bombardier-C-Kram https://gitlab.com/Bombardier-C-Kram

[lurch]

By default Linux will block on executing write(), and wait before continuing until everything is flushed, once dirty page buffers ("write cache") has grown beyond 20% of total memory (changeable through /proc/sys/vm/dirty_ratio ) So only if the system had 40 GB of memory, would there be a chance the whole 8 GB would be in memory.

Ahhh, thanks. Looks like I'm muddling up the way read buffers work (which do cache multi-GB files) with the way write buffers work (which are evidently much smaller!) :upside_down_face:

[Bombardier-C-Kram]

So is there a fix to this problem or like not really? Like can i delete the cashe files somehow?

[lurch]

From what I understand of what @maxnet has said, the "fix" is to simply wait for a while after cancelling the first write, before starting the second write. Are you still seeing progress figures over 100% or has it settled back to normal again? If things are still "funny", have you tried doing a reboot?

[Bombardier-C-Kram]

I’ve rebooted lots of times and waited quite a few hours... I saw it get to 100k percent so idk...I’m just using etcher rn but I just wanted to know if there is a fix for it. I have rebooted a lot of times so far.

Sent from my iPad

On Jun 24, 2020, at 8:06 PM, Andrew Scheller notifications@github.com wrote:

From what I understand of what @maxnet has said, the "fix" is to simply wait for a while after cancelling the first write, before starting the second write. Are you still seeing progress figures over 100% or has it settled back to normal again? If things are still "funny", have you tried doing a reboot?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[maxnet]

I’ve rebooted lots of times and waited quite a few hours...

Is the problem of it going beyond 100% present even if you do NOT "cancel" any writes, but only do a single write session after boot?

What output do you get if you execute "cat /sys/class/block/mmcblk0/stat" twice, once before you start the write, and a second time after the write is complete?

[kwebber3]

I have a similar problem on Windows 11, I restarted the install at 124%. RPI imager v 1.7.2 installing a version from a zip downloaded here. https://downloads.raspberrypi.org/raspbian/images/raspbian-2020-02-14/

[Dustie]

Just adding that I also see this happening in Windows 11, with a OPNsense image and a good USB 3 stick.

[maxnet]

Just adding that I also see this happening in Windows 11, with a OPNsense image and a good USB 3 stick.

Don't think we offer that in the repository. Got a download link?

[Dustie]

Well, the image wasn't actually for the RPI. I just used the RPI Imager as I had it on hand. It was the nano OPNsense image:

https://opnsense.org/download/

However, it worked fine and verified fine too after it reached 200%. Unlike OP I did not cancel anything beforehand.

[maxnet]

Well, the image wasn't actually for the RPI. I just used the RPI Imager as I had it on hand. It was the nano OPNsense image:

https://opnsense.org/download/

Did you extract the .bz2 file first, before opening it in Imager? (we do not compile in libbzip2 support on the Windows platform. So it should normally fail if you open the .bz2 directly. But libarchive may still be able to handle it if it notices you have a bzip2.exe in your path, and can outsource uncompressing to that. Have not tested that though).

Cannot reproduce the issue with Windows 11 under Virtualbox with opnsense-23.1-openssl-nanon-amd64.img (extracted):

[Dustie]

I did extract it first, yes. I can't reproduce it myself either. If I can add helpful input I will of course, but I'm not here expecting a fix or anything. Tool works great. I just found it odd with 200%, googled it, and ended up here :)

[Lopastudio]

same issue here I used HamPi rpi image, and I am running windows 11 22h2.

Edit: It is propably caused by the SDcard being too small. Try using a larger SD card, it might work. I need to try it, but etcher says that my sd card is too small, so it is propably caused by that