pbatard / rufus

The Reliable USB Formatting Utility
https://rufus.ie
GNU General Public License v3.0
29.45k stars 2.61k forks source link

Rufus write-protected my USB drives! #313

Closed ghost closed 10 years ago

ghost commented 10 years ago

hello for first time, i have tried to make an bootable usb drive from an +5GB .iso file. i have used default settings of everything and chose my iso file and usb drive, then i pressed start button. green bar slowly progressed to ~15% and the i got "write protected" message! after that i tried and did repeat the process but now, always i am getting this error at the very beginning! i can't post full log of when this problem happened, because i have had not saved log when this happened, but this is full log of now which give me error at the beginning ....

Rufus version: 1.4.6.440
Syslinux versions: 4.07, 5.10
Windows version: Windows 7 SP1 64-bit
Locale ID: 0x0409
Found USB device '2.0 Flash Disk USB Device' (058F:6387)
Found USB device 'Multiple Card  Reader USB Device' (058F:6366)
Device eliminated because it appears to contain no media
1 device found
Sector Size: 512 bytes
Cylinders: 2000, TracksPerCylinder: 255, SectorsPerTrack: 63
Partition type: MBR, NB Partitions: 1
Disk ID: 0x03663D5F
Drive has a Rufus Master Boot Record
Partition 1:
  Type: NTFS (0x07)
  Size: 15.3 GB (16455303168 bytes)
  Start Sector: 2048, Boot: Yes, Recognized: Yes
Scanning image...
Disc image is an UDF image
ISO label: 'Win7AIO-SP1-x64-Mar2014'
  Size: 5513576448 bytes
  Has a >64 chars filename: No
  Has Symlinks: No
  Has a >4GB file: Yes
  ReactOS: No
  Uses EFI: Yes
  Uses Bootmgr: Yes
  Uses WinPE: No
  Uses isolinux: No
Using ISO: 22in1-Win7AIO-SP1-x64-Mar2014.iso

Format operation started
Requesting disk access...
Caution: Opened drive \\.\PHYSICALDRIVE4 for write access
Will use 'N:' as volume mountpoint
I/O boundary checks disabled
Analyzing existing boot records...
Drive has a Rufus Master Boot Record
Drive has an unknown partition boot record
Deleting partitions...
Could not delete drive layout: [0x00000013] The media is write protected.
Could not reset partitions
Re-mounted volume as 'N:' after error

Found USB device '2.0 Flash Disk USB Device' (058F:6387)
Found USB device 'Multiple Card  Reader USB Device' (058F:6366)
Device eliminated because it appears to contain no media
1 device found
Sector Size: 512 bytes
Cylinders: 2000, TracksPerCylinder: 255, SectorsPerTrack: 63
Partition type: MBR, NB Partitions: 1
Disk ID: 0x03663D5F
Drive has a Rufus Master Boot Record
Partition 1:
  Type: NTFS (0x07)
  Size: 15.3 GB (16455303168 bytes)
  Start Sector: 2048, Boot: Yes, Recognized: Yes

from after that, i can perfectly fine copy the few file that rufus copied to usb drive, from usb to hdd but i can not copy anything to usb, neither i can delete anything from usb or format it!

now even "delete" is absent in right-click menu!

i am getting this damn "write-protected" error from all the programs that i have tried so far! in this picture i was trying to copy a picture file to usb V 303xvz5s (http://ddlw.org/img/303xvz5s.jpg)

after that i thought maybe this usb drive, accidentally in the same time that i used rufus, failed due to hardware problems, so i tried another 16GB usb drive and exactly same thing happened!

so two 16GB usb drives of mine got faulty this way!

how may i correct this? (i am really frustrated.)

AbhishekTripathi commented 6 years ago

0$417. Yesterday I had a 2-month-old USB stick start reporting "no media" so today I bought one of the non-cheap brands you mentioned.

If yours converted to readonly then you can copy your existing data off of it. I can't because mine reports "no media". Fortunately I was using mine as a backup and I still had another backup at hand so I could copy from my other backup onto my new backup.

I had removed the files already from the drive. There is nothing to backup and no file is lost but the usbstick itself is rendered useless. Almost all different ways mentioned on internet has been applied but so far haven't been able to recover either of my sticks.

AbhishekTripathi commented 6 years ago

Below is the log after 5 minutes. This can continue for another hour.

Rufus version: 3.3.1400 (Portable)
Windows version: Windows 10 64-bit (Build 17134)
Syslinux versions: 4.07/2013-07-25, 6.03/2014-10-06
Grub versions: 0.4.6a, 2.02
System locale ID: 0x0809
Will use default UI locale 0x0809
SetLGP: Successfully set NoDriveTypeAutorun policy to 0x0000009E
Localization set to 'en-US'
0 devices found
Found USB 2.0 device 'SanDisk Cruzer Edge USB Device' (0781:556B)
1 device found
Disk type: Removable, Disk size: 7.8GB, Sector size: 512 bytes
Cylinders: 943, Tracks per cylinder: 255, Sectors per track: 63
Partition type: MBR, NB Partitions: 1
Disk ID: 0x001608B7
Drive has a Rufus Master Boot Record
Partition 1:
  Type: NTFS (0x07)
  Size: 7.2 GB (7759986688 bytes)
  Start Sector: 2048, Boot: Yes
Drive has a Rufus Master Boot Record

Format operation started
Requesting disk access...
Opened \\.\PHYSICALDRIVE1 for exclusive write access
Requesting lock...
Will use 'F:' as volume mountpoint
I/O boundary checks disabled
Requesting lock...
Analyzing existing boot records...
Drive has a Rufus Master Boot Record
Volume has an unknown Partition Boot Record
Clearing MBR/PBR/GPT structures...
Erasing 128 sectors
write_sectors: Write error [0x00000079] The semaphore timeout period has expired.
  StartSector: 0x00000000, nSectors: 0x1, SectorSize: 0x200
Retrying in 5 seconds...
Clausinho commented 5 years ago

I just used Rufus for the first time, and it seems to have worked fine. Nevertheless, I wound up reading this forum for some reason, and found it to be one of the most interesting, entertaining, and educational pieces of its type I've ever read. Thanks.

Same. It seems people fail to realize that transfering a rather large amount of data in a short time can brick their cheap usb devices. I am amazed at how logic and calm pbatard handles these issues, even though he is doing this all for the community and not getting paid for it.

elnoxvie commented 5 years ago

Looks like the issue did start from Rufus. I had both thumb drive write protected. Though I appreciate the free software but it causes more problems. The software should probably be taken down in order not to brick any more thumb drive.

pbatard commented 5 years ago

@elnoxvie, you do understand that what you report is what one can expect to happen when using fake drives (i.e. USB flash drives that have had their firmware modified to pretend they have more capacity than they really have).

Hundreds of millions of people, who aren't taking to the internet to demand that the software be taken down, are proving, day after day, that Rufus does not brick drives.

So I'm afraid a handful of reports, provided by people who, and if you want to dispute that assertion, I'll be very happy to have you post to demonstrate otherwise, don't seem to understand how writing to an USB works in Windows, and who haven't started to provide even the beginning of and explanation as to how Rufus could damage drives (whereas I have repeatedly tried to demonstrate, technically, how it cannot), are simply not going to cut it.

From now on, the only replies I will accept on this thread are ones that contain some technical details. So, as I already repeatedly asked folks who believe that Rufus has a bug that bricks drives to do, please get in touch with the manufacturer of the drive so that they replicate the issue, and then have them post technical findings that disprove the ones I provided earlier. Surely, if there is software that causes drives to be RMA'd, they should be more than happy to demonstrate that said software has a flaw so that they don't have to spend their money on RMA. But of course, the fact that, after years of "But Rufus so has a bug that destroys drives" posts, there still hasn't been a single flash drive manufacturer out there who posted anything on this allegedly blatant issue with the software is clear evidence that, contrary to what some people here still want to believe, whatever caused some drives to be bricked had nothing to do with Rufus (such as, for instance, unknowingly having purchased a bunch of cheap "fake" drives, that WILL get bricked as soon as you write more data on them than they actually can hold).

elnoxvie commented 5 years ago

Hey, if they are enough report, Why are you denying it so hard. When unetbootin or others don't generate such issues. It must be the method that you used is not safe. It is not like the incident happened on one memory card only but two. Maybe I got unlucky. Just trying to highlight to people the method you used in Rufus is unsafe and could trigger the write protected.

Given how many of such issues have been submitted, it should be at least a good indication that the software has bugs. As a developer myself, I am actually being polite here.

Sorry for my ranting but people please keep away from Rufus for now if you don't want your thumbdrive getting write protected and probably irreversible process.

nxdiamond commented 5 years ago

"you do understand that what you report is what one can expect to happen when using fake drives"

No, fake and modified drives are not this problem. Cheap design and inadequate testing by genuine manufacturers is the problem. Writing 2GB of data at USB3 speed can overheat the internal electronics of a poorly designed flash drive.

Mr. Batard it is not your fault. It happens to you a lot because your program writes 2GB of data at USB3 speed. Or if someone's writing a Windows ISO then your program writes 4GB of data at USB3 speed.

I've returned around 5 USB drives during the past 3 months. One time a store employee took photos of his PC's Windows screen reproducing the failure, and phoned the Japanese distributor of the USB sticks (the same distributor handles at least two internationally famous Taiwanese brands).

pbatard commented 5 years ago

@elnoxvie, I'm seeing a lot of issues with write protected drives that have nothing to do with Rufus on superuser. So clearly, as opposed to what you stated, other applications and usage do generate these issues as well...

Also, as I've explained in earlier comments, your "how many such issues have been submitted" is pretty much what I'd expect from coincidental failures from a software that is downloaded more than 3 million times every month (which, I suspect, is likely to be a lot more than unetbootin). In fact, given the amount of fake drives that I know are being sold to unsuspecting users, as well as some known occurrences of flash drives (e.g. some PNY models) having firmware issues that will cause them to fail, I am pretty confident that the amount of "your software killed my drive" reports I am getting on this issue tracker or elsewhere, is actually quite low considering the amount of people who must be using Rufus with fake/low-quality/problematic drives, and BELIEVE that because their hardware happened to fail while they were using it with Rufus, then Rufus must be the issue...

Also, since you're a developer, then please feel free to poke holes in my previous explanation that, no, even if it had a major bug, Rufus can absolutely not damage an USB drive, unless the Windows API and the Hardware Abstraction Layer also have a major bug. There's really no in-between here: Either my explanation is correct, and therefore there's ZERO possibility that Rufus can send the alleged "self-destruct" command that a handful of very vocal people, who don't seem to understand how partitioning/formatting/file-copying onto an USB drive work, are desperately trying to put forward. Or, my explanation is wrong, in which case it should be exceedingly easy, for a developer with some knowledge of the Windows APIs as well as USB Mass Storage, to disprove it, even more so as the whole source code of Rufus is also easily accessible.

TL;DR: When there exist hundreds of millions of people using a software, then it is completely expected that a handful of people out of these millions users will experience coincidental failures, that have nothing to do with said software (yes, even on multiple drives, which can be explained by fake/low-quality/buggy-firmware, all of which are real hardware issues that there exist numerous examples of), and that they will be inclined, sometimes very vocally, to want to point the finger at the software. Yet, if that handful of user reports is all the "evidence" there exist, then, as much as the people who experience the failures would like to pretend otherwise, these reports are still dramatically inadequate to be accepted as "evidence", even more so if none of these reports are able to provide any detailed technical data that can disprove the "coincidental failure" factor.

nxdiamond commented 5 years ago

The Windows API and HAL (and, more importantly, drivers) can't damage a USB drive that way either. The drive damages itself because of poor design by manufacturers.

elnoxvie commented 5 years ago

@pbatard Well let me put it in this perspective.

Few days ago, i used the 4gb sand disk cruzer. (SDCZ36-004G) and create a bootable disk using etcher. It's works fine.

And just yesterday, i tried using rufus to create a bootable, gpt-uefi and it works. files are intact but not bootable, immediately i tried to reflash it another time and got the write protected issue.

Note: both memory card contains no bad sector after running a disk check.

on the same day, i used my sandisk microsdxc 16gb and format it with rufus, same thing happened. It got write protected. I understand they are other factors that could cause the drive to malfunction, such as usb driver in the OS or poor usb drive, usb electricity or fake usb drive.

so let's look at certain scenario. 1st. i believe both my memory card are from reputable company and bought from local reputable shop so this shouldn't be counted as the root cause. 2nd. etcher and rufus are both of the same nature but etcher didn't cause such issue to occur but rufus did. Presumably, there are certain codes in rufus aren't safe enough for certain drive to work coupled with multiple external factors, bad usb plug(PC drive), bad sectors etc either of these triggered such issues. 3rd. i also suspect the issue were caused partially by bad usb panels from those desktop cases. (yes i used a diy computer with cooler master cases) and drive were plugged in usb on the front panel during formatting. It may be caused by bad or cheap usb panel. If this is an issue, maybe there are room for improvement for rufus whereby it could imitate other software such as unetbootin or etcher as not to trigger such issue.

It would be good to provide an alternative way of writing the files to the usb. Presumably if the root cause were the codes that tried to achieve high speed transfer then maybe a safe method could be made available. ( just a speculation).

What i theorize is rufus achieved amazing speed with some low level api that may trigger such issues to happen. Of course, this is just my speculation and the issues here is beyond my knowledge to investigate.

anyway, i am grateful for the free software and I want rufus to success but the problem here is severe enough since this write protected seems to be irreversible process so it would be good to spend sometime to investigate on the issue.

if you need user's help, i believe we will more than willing to lend a hand to provide logs or os specs etc.

Thank you for listening.

pbatard commented 5 years ago

and got the write protected issue.

both memory card contains no bad sector after running a disk check.

Hold on. If you ran the bad block check after you got the drive write protected, then clearly the drive is not write protected, since you can not validate that a drive is fine without writing each individual block (which is the whole idea behind a bad block check).

1st. i believe both my memory card are from reputable company and bought from local reputable shop so this shouldn't be counted as the root cause.

You'd be surprised at how far bad or flaky hardware manages to make it up the "reputable store" chain. At the end of the day, people who distribute flaky hardware always manage to undercut the pricing of people who distribute reputable one (since they don't have to care about quality), and obviously, since the retailer's goal is to maximize their profit, they tend to go with the cheapest bulk offer, especially as it's impossible for a retailer to validate the quality of all the products they resell.

In other words, if you want to theorize that your hardware was fine without having any means of providing concrete evidence (result of a bad block check ran before you used the drive), don't be surprised if I theorize that your hardware wasn't that fine.

2nd. etcher and rufus are both of the same nature but etcher didn't cause such issue to occur but rufus did.

Etcher is a 'dd' writer. Rufus isn't. That's why for instance Etcher is hopeless for writing Windows ISOs. So you're wrong in your assumption that Etcher and Rufus are of the same nature. Also, punctual evidence that Etcher worked where Rufus failed is hard to see as anything but coincidental occurrence, especially as one is copying blocks and the other is copying files. A much more meaningful test, if you want to assert that Rufus destroys drives where Etcher doesn't is make sure that you use the same image in both cases, and also make sure that, when Rufus prompts you, you select 'DD mode' and not the default 'ISO mode'. Then you will have software performing operations that are of the same nature. Also, make sure you do that from 2 different computers, as I could bring up flaky extension cables or bad connections as the reason for the failure when using Rufus, especially when working with USB 3.0 hardware, which is sensitive to bad connection and interferences.

Presumably, there are certain codes in rufus aren't safe enough for certain drive to work

Nope.

You know, I am starting to get a bit tired of people, who clearly haven't looked at the code, or who don't have the capacity to understand the code, ASSUMING that Rufus is doing something very hackish when copying files. THAT IS NOT THE CASE

We're using CreateFile() and WriteFile(), just like File Explorer does. And we don't apply tricks like running multiple threads in parallel to try to speed things up. Heck, we're not even using OVERLAPPED to run these operations asynchronously. So, when you are accusing Rufus of doing something unsafe, then you are effectively accusing Microsoft of doing something unsafe in their OS, because CreateFile() and WriteFile() are the most basic and common file I/O operations Windows OS is expected to run.

If you believe that there exist ways to use those in a unsafe manner when simply writing files to a drive (which is what Rufus does), then please open an issue with Microsoft, because I'm pretty sure they will be very interested in hearing how these APIs have a major reliability bug, that has gone undetected for 30 years (since that's about how long these APIs, which again are the most basic ones you can use for File I/O, have existed).

3rd. i also suspect the issue were caused partially by bad usb panels from those desktop cases.

Interesting...

whereby it could imitate other software such as unetbootin or etcher as not to trigger such issue.

I believe I already said that elsewhere, but I have zero interest in slowing down operations for everybody, on account of a handful of people using flaky/unreliable hardware. You use bad/flaky hardware, you pay the price. And if, as a result, you want to point the finger at the developer of some software that had nothing to do with it, I guess I'll have no choice but to continue handle that. But I'll also continue to point out that the issue could only have come from the hardware.

It would be good to provide an alternative way of writing the files to the usb.

Brilliant! Now you are effectively asking me to ditch the standard Windows APIs, to hack my own, with all the problems and reliability issues that will incur. Do you want me to write my own OS as well?

What i theorize is rufus achieved amazing speed with some low level api

<sigh>. Please get somebody who understands C and has some level of familiarity with Windows look at the Rufus source (Hint the copying of files is handled in either udf_extract_files() or iso_extract_files() in iso.c).

This "Rufus must be using some hackish low level tricks", which can EASILY be disproved by looking at the source (which again is completely public) has got to stop. Rufus does NOT use any low level API to speed up the copying of files. Rufus does NOT even write individual blocks or send USB commands on its own while copying files. Instead, ALL Rufus does is tell Windows: "Create a file on drive X:" and then "Here, write this data to the file you just created", using the super standard CreateFile() and WriteFile() APIs (which are in effect so limited that you cannot even pass any options that could even be considered dangerous - but there again, feel free to look at the very official documentation and dispute my statement, if you believe that any of the super common options used by Rufus should be considered "unsafe").

since this write protected seems to be irreversible process

What does Linux say about that? I'd really like to see all the people who experience the issue try to recover the drive from another OS. For instance, I know that some ChromeOS images will make Windows go a bit berserk when attempting to repartition the drive (pure Windows bug, for which Rufus actually had to add a workaround), but you can easily recover such a drive from Linux.

if you need user's help, i believe we will more than willing to lend a hand to provide logs or os specs etc.

I ALWAYS expect to see FULL logs from Rufus if you experience any issue. As a matter of fact, I am surprised on how much of the new entries for "Rufus write protected my flash drives" seem NOT to want to provide a log. Surely, if you experienced a failure, and tried with another drive, the first thing you should be cautious to do if you experience a similar failure with that second drive, would be to save the full log, so that you can provide it to the developer of the software you believe is causing the issue. Or maybe that's just me. Heck, almost none of the reports above even provide comprehensive data about the flash drive or memory card that was used (which the Rufus log would provide, especially the VID:PID). I made sure that Rufus could provide a very comprehensive log of its operation, which is also very accessible (just click the last small button left of START). Yet, everybody seems to come to this post on the false assumption that this is a super common issue (it isn't), and therefore that they don't need to provide ANY details on the very wrong assumption it should affect such a large number of people that the developer also should have no issue reproducing it.

So let me tell this for all the people who might want to add their own story to this issue: you shouldn't be "willing to provide logs or specs". If you want to add any new data on this issue, then you should provide them from the get go. You should also provide the full details of where exactly you purchased your drive and when. If you are confident that Rufus is the culprit, then you should have the confidence to post all the data required to try to prove it (so that we can isolate bad batches, disreputable stores, cheap drives and so on, which you definitely want to remove as possible factors if you are hell bent on pointing the finger at the software).

nxdiamond commented 5 years ago

Etcher destroyed two of my USB sticks, but it's not the fault of etcher, it's the fault of defecitve makers who don't design or test their USB sticks to prepare for the heat generated when writing 2GB or 4GB at USB3 speed. In the etcher forum I suggested that a favour could be done for victims who buy USB sticks from incompetent manufacturers. Maybe after writing each 100MB of data, pause for 30 seconds to let the USB stick's internal electronics cool down. The maker of etcher doesn't want to do that. Who could blame him? Stupid manufacturers are stupid manufacturers.

PhoebeAnn commented 5 years ago

nxdiamond might have something there. Add an optional config parameter to Rufus to add "cooling-off periods" when writing large amounts of data at USB3 speeds. Good luck determining how long and how often these periods should be.

pbatard commented 5 years ago

Add an optional config parameter to Rufus to add "cooling-off periods" when writing large amounts of data at USB3 speeds.

Nope. Not gonna happen.

First of all, this "some drives might need cooling off" is complete speculation and I don't apply fixes for speculative issues. I apply fixes for actual issues that are attached to complete verifiable technical explanations. Furthermore, even this speculation is assessing that the hypothetical "real" issue has to do with poorly engineered hardware, and not Rufus.

Again, considering that hundred of millions of people have been using Rufus with all sorts of drives and memory cards, including poor quality / overheating ones, and that, when looking at it objectively, one can only conclude that the exceedingly small amount of users that are reporting flash media issues would be called "statically expected noise" in any scientific experiment using the same sample size as the number of usages of Rufus, it is very hard to see any justification for adding a "cooling off" option.

Contrary to what the people who appear to have been affected with the "write only" issue have been trying to assert, this is not a widespread problem. It is the expected noise stemming from a bell curve where the vast majority of people do get drives that are reliable enough to sustain the kind of standard operations that Rufus and other software (including Windows File Explorer) require from the hardware, but a small amount at the low end of the curve don't. I therefore see no reason to try to make unreliable hardware, that is going to fail one way or another anyway, fail somewhat later. We're not pining for "fail better" here.

johnheintz commented 5 years ago

I think hundreds of millions of users is also very speculative. Can you send the data to back that up.

Sent from Mailhttps://go.microsoft.com/fwlink/?LinkId=550986 for Windows 10


From: Pete Batard notifications@github.com Sent: Sunday, December 16, 2018 7:47:54 PM To: pbatard/rufus Cc: johnheintz; Comment Subject: Re: [pbatard/rufus] Rufus write-protected my USB drives! (#313)

Add an optional config parameter to Rufus to add "cooling-off periods" when writing large amounts of data at USB3 speeds.

Nope. Not gonna happen.

First of all, this "some drives might need cooling off" is complete speculation and I don't apply fixes for speculative issues. I apply fixes for actual issues that are attached to a complete verifiable technical explanations. Furthermore, even this speculation is assessing that the hypothetical "real" issue has to do with poorly engineered hardware, and not Rufus.

Again, considering that hundred of millions of people have been using Rufus with all sorts of drives and memory cards, including poor quality / overheating ones, and that, when looking at it objectively, one can only conclude that the exceedingly small amount of users that are reporting flash media issues would be called "statically expected noise" in any scientific experiment using the same sample size as the number of usages of Rufus, it is very hard to see any justification for adding a "cooling off" option.

Contrary to what the people who appear to have been affected with the "write only" issue have been trying to assert, this is not a widespread problem. It is the expected noise stemming from a bell curve where the vast majority of people do get drives that are reliable enough to sustain the kind of standard operations that Rufus and other software (including Windows File Explorer) require from the hardware, but a small amount at the low end of the curve don't. I therefore see no reason to try to make unreliable hardware, that is going to fail one way or another anyway, fail somewhat later. We're not pining for "fail better" herehttps://en.wikiquote.org/wiki/Samuel_Beckett#Worstward_Ho_(1983).

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/pbatard/rufus/issues/313#issuecomment-447706535, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AdoduUnFubcoYxr5x0sytg_Z7cpW1siNks5u5wXagaJpZM4Byyzu.

pbatard commented 5 years ago

I think hundreds of millions of users is also very speculative. Can you send the data to back that up.

Here are the current number of downloads for Rufus 3.4 (which was released 11 days ago):

Rufus-3.4.appx: 1078 downloads
rufus-3.4.exe: 688862 downloads
rufus-3.4p.exe: 255989 downloads
rufus-3.4_arm.exe: 776 downloads
rufus-3.4_arm64.exe: 2004 downloads

ANYBODY can verify these stats, as github lets you see the number of downloads from any project that uses github releases (search for something like gdc), and not just only the owner of that project. This is actually a bit lower than usual due to the fact that the automated update check got broken between 3.3 and 3.4. But even if you don't believe me, with the people who choose to download older releases (not shown in the above) along with people downloading Rufus from unofficial mirrors, you should reasonably consider that Rufus sees at least 3 million downloads/month, or 36 million downloads a year.

Now, it is unlikely that all of these downloads are one time use, so there has to be some >1 factor when estimating the number of uses. And of course, Rufus has not always seen that high a number of downloads, but then again, it has been publicly available since 2012.

I therefore consider a realistic estimate that there has been at least 200 million uses of Rufus. Or hundreds of millions.

elnoxvie commented 5 years ago

@pbatard Well, My suggestion is to put up a list of thumb drives certified to work with rufus. This could prevent users from damaging their so called flaky thumb drive. Plus, it's not like every users know what thumb drive is A OK with rufus or not.

If users were to do a trial and error on their usb thumbdrive ( since the damage is irreversible) and guess what? rufus is to blame. WHY? because this only happened after they use rufus. Despite all the technical correctness or claimed by the author or whatsoever. People will look at it that way.

Therefore, a little warning or lists of thumbdrive certified to work would be good. Otherwise, they will risk damaging their little flaky thumb drive.

The certified USB list, could also act as a disclaimer here.

Brilliant! Now you are effectively asking me to ditch the standard Windows APIs, to hack my own, with all the problems and reliability issues that will incur. Do you want me to write my own OS as well?

Woah. hold your horses. I am not implying you to do so. I am just saying there could be a reason why after their formatting using rufus, the issue occur. So maybe, there are issues with methods that were used.

e.q. Rufus used copy files instead of DD and unetbootin using DD only. Thus, something can be implied from there. Maybe copying files at high speed damaged the flaky usb drive.

Anyway, i understand it's not easy to pinpoint the issue here. Maybe you could consider my suggestions above (regarding the usb list) as to avoid users from damaging their so called FLAKY thumb drive.

elnoxvie commented 5 years ago

@nxdiamond What thumbdrive are you currently using? can you recommend some that you think work well with rufus?

pbatard commented 5 years ago

Maybe copying files at high speed damaged the flaky usb drive.

In that case, you will need to tell people to NOT use Windows File Explorer either, because Rufus does the exact same thing and (unless you have some evidence to demonstrate otherwise), Microsoft does not add "cool down" in File Explorer when detecting a specific set of USB 3.0 drives, or any other drive for that matter.

I am just saying there could be a reason why after their formatting using rufus, the issue occur.

And I have repeatedly tried to provide evidence, which nobody seems to want to look at (because God forbid someone backs up their claim with code) why what you assert about the Rufus code doing something "special" that may trigger a hardware failure is bullshit.

lists of thumbdrive certified to work would be good.

Feel free to create a web page with that information.

Or are you expecting the developer to go and buy every flash drive under the sun, and invest a lot of time maintaining the list you suggest. And how about you ask the same for the Etcher or Unetbootin people as well? After all, since we seem to (somewhat) agree that the issue must be tied to the hardware, there's certainly no logical reason not to ask the same from them (unless you want to dispute the earlier report that some people have their drives fail while using Etcher).

Anyway, i understand it's not easy to pinpoint the issue here.

That's because there's no software issue.

Anyone familiar with Windows APIs, the Windows Hardware Abstraction Layer, C, and who has spent a few minutes looking at the file copy code from Rufus will be able to tell you that. But the problem, which seems to be a very general issue today and one that is not limited to only the computing world, is that, as soon as something requires placing some trust in somebody who has expertise you lack (and please understand I'm not talking about myself here - I'm talking about people who are able to validate or invalidate claims made by somebody from the same field), then, in this age of trying to make yourself look smart in a comments' section, it somehow seems to become fair play to consider that expert advice can be bypassed altogether, and that anyone's non-expert assertion should be considered equally valid as the experts'.

When you are familiar with the fields I have mentioned above, it is actually very easy to pinpoint that, logically, the issue cannot be with the software, which, as the software developer, is all I require (as I have no interest in pursuing the matter if I can demonstrate that it has nothing to do with my code). But of course, as soon as you make that claim, you get bombarded with "oh yeah, like we're going to trust a software developer not to have a vested interest in pretending that their software is not responsible". Quite frankly, since this is basically doubting my integrity as a software developer, without being able to bring a shred of evidence why it should be doubted, it is starting to get exceedingly tiresome.

If you want me to lock this thread and prevent anybody from posting any further (which, as you have perhaps failed to notice I have not done in the 4+ years this issue has been opened), as well as shut down any attempts to create a new one (and not give a damn about expected cries of "developer conspiracy" that will result), you are certainly doing a good job...

elnoxvie commented 5 years ago

Or are you expecting the developer to go and buy every flash drive under the sun, and invest a lot of time maintaining the list you suggest. And how about you ask the same for the Etcher or Unetbootin people as well? After all, since we seem to (somewhat) agree that the issue must be tied to the hardware, there's certainly no logical reason not to ask the same from them (unless you want to dispute the earlier report that some people have their drives fail while using Etcher).

Woah. why do you have to imply everything in a negative way? i am saying this can be done via crowd sourcing. Create a wiki page and let people chip in.

In that case, you will need to tell people to NOT use Windows File Explorer either, because Rufus does the exact same thing and (unless you have some evidence to demonstrate otherwise), Microsoft does not add "cool down" in File Explorer when detecting a specific set of USB 3.0 drives, or any other drive for that matter.

True enough but how is that copying via windows file explorer to thumbdrive doesn't cause such issue and only when using rufus, the issue occur. This left me baffled. Just saying.

elnoxvie commented 5 years ago

@pbatard Reading back on my previous comments. I kinda feel sorry. Sorry for my earlier remarks on taking down the software. Clearly, i was not thinking clearly and spout some nonsense there. guess, i wasn't thinking rationally as two of my thumb drives just got bricked.

Anyway, hopefully this is as you said the issue with the hardware. Will try rufus again, if i could get my hand on a new thumbdrive under warranty.

Thanks for taking your time to explain the issue.

nxdiamond commented 5 years ago

"What thumbdrive are you currently using? can you recommend some that you think work well with rufus?"

I don't think it would be fair for me to answer that. For example there is a famous Taiwanese brand that used to make USB 2 flash drives and I never had a problem, but their USB 3 flash drives failed and a retail store let me exchange for other brands. One other Taiwanese brand is less famous, and their USB 3 drives lasted a bit longer before failing, but still failed. The retail store let me exchange one for a third brand (a Japanese brand and I paid the difference in price), and they refunded my money for one because I'd already bought a fourth brand while travelling (a US brand but really made by a Japanese company). I do not know yet if the latest one is going to fail, but someone else reported a failure with the same brand.

I'm going to go out on a limb and speculate that one Japanese brand deliberately limits the speed of their USB 3 flash memories to be about half the speed of the competition, and this might be why they haven't failed on me.

There's another brand of USB 3 flash memory that I've been using for a few years now. This one has a case made of metal instead of plastic. It runs at full USB 3 speed and hasn't failed even after a few years.

So this is why I bet the failures are due to overheating of the internal electronics.

===

"In that case, you will need to tell people to NOT use Windows File Explorer either"

Yes indeed. Using Windows File Explorer in Windows 10, I could pause a copy operation. Every 30 seconds I tried to pause the writing of a bunch of files, wait 30 seconds, and resume. Windows 10 was slow responding to my clicks on the pause button. The USB stick died shortly after that.

elnoxvie commented 5 years ago

@nxdiamond thanks for the insight.

pbatard commented 5 years ago

Create a wiki page and let people chip in.

Sorry, when it comes to propping software or hardware, I want to provide information that I have verified myself. That's the reason why you don't see a wiki page on ISOs that are compatible with Rufus, because I am confident that, as soon as single person experiences some unrelated issue with an ISO, they will be tempted to make it their life crusade to have that ISO removed from the wiki, even if the issue they had was unrelated. And I'm sure we'd see exactly the same for flash drives ("I've had a coincidental failure with this flash drive, therefore I feel that I am entitled to say that ALL flash drives of this make should be removed from the list").

As a matter of fact, what I am highlighting above with overgeneralization is exactly the problem I'd like people who want to post in this thread to be mindful of.

Nobody is denying that you may have seen repeated drive failures while using Rufus, and, of course, everybody understands why it would seem logical to want to point the finger at the software you were using while you experienced these failures (or, even if you agree that the software wasn't directly responsible for these failures, want the software to take into account the possibility of hardware failures occurring while it is writing data). However, you have to be exceedingly careful about trying to generalize an issue you have experienced and promote the idea that, because you have seen it once or twice, under conditions that can't help but have specific components related to your environment, then it can be concluded that it "must" be affecting a significant number of people.

Now, the following is really a rule that I wish everybody who opens an issue in this tracker or contacts me by e-mail would follow, because it doesn't just apply to what we're discussing here:

Until proven otherwise, you should always assume that you are the ONLY person to experience the problem you report, and therefore ensure that you provide all manner of details, that are specific to your environment, the hardware you used, and the operations you conducted, to the person you report it to; as if the troubleshooting task was to find what, in your specific environment or usage, is the cause of the problem.

Which brings me nicely to:

how is that copying via windows file explorer to thumbdrive doesn't cause such issue and only when using rufus, the issue occur.

This is the part where you are overgeneralizing and assuming that, because you have seen a problem while using Rufus and not while using File Explorer, there doesn't exist a similar or greater number of people, who aren't using Rufus, and who might have seen their drive fail in the same fashion while using File Explorer. The problem here is that you are not in effect providing a receivable proof that File Explorer is safe whereas Rufus isn't. All you have is punctual evidence that, in your environment, you happen to have seen drives fail while using Rufus, whereas you haven't seen the same while using File Explorer.

Now, if, out of the millions of users who are using Rufus on a daily monthly basis ([EDIT] sorry for the typo that makes it look like I am overgeneralizing myself — I actually meant to write monthly and it doesn't diminish the point I am making anyway), I was seeing a significant number of reports (most of which would be very public, on this issue tracker, or on public forums like superuser), that there exist a large number of drive failures during Rufus usage, then I would take what you highlight above as contributing to undeniable evidence that Rufus is indeed causing drive damage. But the problem is, I am not seeing that at all. In light of that, and in light of the statistical noise that is expected from an application that accesses flash drives of various reliability and that sees high volume usage, I therefore can't help but find your alleged evidence that Rufus is more inclined to cause drive failure than File Explorer, as well as the previous allegations from the 4+ years this issue has been opened, as exceedingly inconclusive. Even more so when I can vouch that the operations that are being conducted when Rufus is copying files have nothing susceptible of causing more drive failures than when using File Explorer.

Sorry for my earlier remarks on taking down the software.

I appreciate that. And please understand that, even though I feel it is my duty to try to dispel the idea that Rufus is causing drive damage, which a stance that some might consider intractable, I can totally understand where you are coming from. But there again, you (and others) need to be mindful of overgeneralizing an issue you have faced, unless you can back up, with concrete evidence, why it should indeed be considered a general problem (which is the precise reason why you'll see me request that people RMA the drives and get in touch with the manufacturer, so that, if the manufacturer has reason to believe the software had any part in the failure, they will be able to provide technical and detailed evidence of it, which will be a lot more receivable than overgeneralized punctual occurrences).

Will try rufus again, if i could get my hand on a new thumbdrive under warranty.

Great. Please make sure that, if you do that, and if you do experience another hardware failure, you do save the full log and provide it to me. Also, please make sure to run a complete bad blocks check on your drive. Note that Rufus can do that as part of its operations (and the nice thing is, if you do it from Rufus, then the result of the bad blocks check will appear in your log), but if you don't trust the bad blocks check from Rufus, you can also use an external application. You can select to run a bad blocks after you click Show advanced format options. Note that, if you want to run a comprehensive bad blocks check, you may have to find from the manufacturer of the drive/memory card if the flash memory used is of type MLC, TLC or SLC, so that you select the right option in Rufus. Also note however that a 4 pass bad block check may take a very long time...

Sopor commented 5 years ago

Balena destroyed my USB drive ! How to recover? Etcher is corrupting SD and USB drives Etcher destroyed 2 of my USB sticks, but I bet it's not etcher's fault UNetBootin destroying USB drives USB pen drive not recognized after using UNetbootin

lurch commented 5 years ago

@Sopor- Yeah, for some reason people always jump to immediately blaming the software they were using, rather than their flaky / broken / fake hardware :-/

@pbatard From an ex-Etcher-developer, you have my sympathies!! See also https://github.com/raspberrypi/documentation/issues/1042

pbatard commented 5 years ago

@lurch, thanks for the support. And since you were involved in that project, let me just say that, from what I have seen of it, you guys did a very fine job with Etcher... 👍

And yeah, since it's gotten popular (for good reason) I am statistically expecting Etcher to run into the same kind of user reports that Rufus gets when it comes to people thinking that the application is responsible for whatever drive failure they experienced. So it does feel a bit less lonely to get confirmation that reports of "Application X killed my drive!!" are far from being limited to Rufus.

I also hope that seeing this this will help temper people who are absolutely certain that Rufus (or any other application that writes to flash memory based media for that matter), has the means to destroy their drives...

unsungNovelty commented 5 years ago

https://github.com/pbatard/rufus/issues/313#issuecomment-322045700

Thanks a lot for your explanation @pbatard! My intention was never to complain about the bug (thought it was a bug from your end), but just to report it and possibly point out if there is a chance the issue is in Rufus and you overlooked it (which looks to be wrong). Looks like I am clearly wrong and sorry to have come off as the end user who just came to the project to complain about it. That was never my intention. Also looking back at the comment, I apologise if the language wasn't decent too!

I've used Rufus before this and after this incident (although to be frank, a bit cautiously after the incident. Which just means fingers-crossed and hope the USB won't get messed up :smile: ) and it has been reliable during the limited instances I've used it. But couldn't fix the broken USB though. Anyways, thanks a lot for your work on this project and I really appreciate it. :)

If you're ever in Bangalore,India for a tech conference or something, a beer/coffee is on me! :beers: :coffee:

daddyslittleone commented 5 years ago

Rufus also made useless my 64GB Stellar Flash drive! I didn't save the log and I will not redownload your nefarious software to make a new one. These are most certainly NOT isolated incidents, a simple Google search returns hundreds of results with people looking for help with this exact same scenario on multiple forums! Fix this issue or remove the software! It's that simple!

pbatard commented 5 years ago

These are most certainly NOT isolated incidents

As had been previously explained, these incidents fall in the range of coincidental failures one would expect according to how widely software like Rufus and Etcher are used, as well as the prominence of "fake" USBs and so on...

Of course, you're free to reject this explanation. But if you think that there is some kind of major flaw in the software, that is entirely Open Source (and, to be honest, not that complex at all when it comes to writing data to a drive, since Windows severely limits what we can do there) and therefore one where it should be very easy for any specialist or advanced user (such as USB flash drive manufacturers who of course should have a VERY REAL VESTED INTEREST in seeing software that destroys drives be put out of circulation) to point to such flaw, then, after the software developer has repeatedly debunked this possibility with facts, you should understand that, if you still choose to believe that there is a conspiracy to ignore a "widespread" issue, the burden of proof is on you. It's that simple!

lurch commented 5 years ago

@daddyslittleone

Rufus also made useless my 64GB Stellar Flash drive! I didn't save the log and I will not redownload your nefarious software to make a new one.

That kind of attitude isn't very helpful. Even if there was a problem (and I agree with @pbatard that there isn't), without co-operation from people like yourself who are experiencing problems, it's impossible to investigate further.

Unfortunately it seems that most people who are quite happy to complain and accuse Rufus or Etcher of "breaking their drives" are almost always unwilling to provide further assistance :disappointed:

mrk310310 commented 5 years ago

My 2 gb sd card just got Rufused :). Reporting in. Is there any UnrufusMySdCard.exe? Probably some sdcard/usb drives don't like the process.

pbatard commented 5 years ago

I suggest you validate that your card really can't be used from a Linux host. You may also want to look what Device Manager / Disk Manager report in Windows.

For instance, I know that some Chromium images with tons of GPT partitions will completely throw Windows off balance (pure Windows bug) and make it look like the card is inaccessible, whereas if you repartition and reformat the card in Linux, you'll find that it works just fine, and you will be able to reuse it from a Windows host.

Now, if even Linux can't access the card, then you will have to RMA it to the card manufacturer.

Also, it always help when people who want to report flash storage issues do provide the full information about their media (brand, complete type, when it was bought, from which store, etc.) as well as the VID:PID reported by Rufus in its log when accessing the media (which would be the VID:PID of the adapter when using an SD card). And of course you should also report the kind of image you were trying to write on it.

Else, it's pretty much akin to telling a car mechanic "My car doesn't work", over the phone, without reporting the car model or its age or any other detail that can provide insight as to the reasons for the breakage, such as how exactly you were trying to use the car when you encountered the issue...

Sopor commented 5 years ago

@mrk310310 You can always try one of the suggested program i mentioned here

mrk310310 commented 5 years ago

I wasn't asking for help, just reporting. The image was a 600mb Win10-live-usb thing, "burned" with GPT option, on Windows 7. The sd card was - Kingston 2gb microsd SD-C02G Japan, on the other side of the card were two lines - 0832 U59502Y.

Also I don't want the dev to feel bad because of this situation. Again, some memory chips probably just work weird.

Sopor commented 5 years ago

Is there any UnrufusMySdCard.exe?

@mrk310310 Isn't this to ask for help?

PhoebeAnn commented 5 years ago

I found the only way to deal with a "bad" usb stick was with the diskpart utility in Windows. In my case, a GPT had been created from a FreeNAS ISO that was unrecognized and unbootable in either a Windows or Linux machine. Disk manager saw it, but couldn't do anything useful with it.

yodies commented 5 years ago

I just had two drives destroyed, one after the other. One a 16GB SanDisk Cruzer Glide, the other a 16GB Sony thumbdrive. Both of them failed saying that explorer.exe had a lock on the disk. I closed Explorer, and then the thumbdrive got marked as write protected. Two failures in a row is not a coincidence. I now have to go buy TWO 16GB drives for coworkers.

pbatard commented 5 years ago

One a 16GB SanDisk Cruzer Glide, the other a 16GB Sony thumbdrive.

Where and when did you purchase them? Can you RMA them?

Last time I checked, SanDisk had a pretty comprehensive RMA policy.

Also, please try to repartition/reformat your drives in Linux.

I know of some images (e.g. Chromium) that, because of a pure Windows bug, will make it look like the drive cannot be repartitioned or reformatted in Windows after the image has been written, whereas Linux or MacOS have no trouble doing so.

Two failures in a row is not a coincidence.

Usually not. However it also doesn't not mean that the application you were using at the time was the cause of the issue.

For instance, some FreeBSD images will instantly make Windows 7 or earlier BSOD, regardless of the application being used to write the image (another Windows bug - You can literally create a USB flash drive that make any Windows 7, Vista or XP machine crash as soon as it is plugged). But of course, when you experience a BSOD while you were using a specific application to write data, it's easy to think that it's the application that caused the system crash, even if it had nothing to do with it.

If you can RMA the drives, I would strongly encourage you to do so. And if, despite all I have explained above, at length, on how when you are familiar with the manner Windows lets application write data to a drive, and how USB Mass Storage operates, then the only way a Windows application that only uses the generic abstracted I/O API to write to a drive could ever destroy a drive is if there also existed a major Windows bug, you still refuse to consider that your issues could have to do with something else than Rufus, then I will encourage you to let SanDisk or Sony support know that you were using Rufus at the time, and ask their specialists whether they believe that this software (of which they should be able to also consult the source) could be what actually caused your drives to fail. If these specialists think that there is a flaw in the explanations I have provided in this thread, with regards to how it is just impossible for Rufus (or other generic image applications like Etcher), to destroy drives in the manner that people who have experienced failure(s) allege, then they should have little trouble demonstrating, with indisputable proof that it is indeed possible for an application that only issues WriteFile() to write data to an USB Flash drive, and that only does so against the Physical or Logical drive that abstracts the device, to actually send some kind of "self-destruct" commands.

And yes, I understand that, because you have seen a drive failure twice in a row while using Rufus, it is tempting to believe that the developer would have a strong interest spewing bullshit, while refusing to acknowledge a real issue. But I can assure you that, if there was a real issue here, and if what I've been saying about abstraction and the handling of USB flash drive in Rufus making it impossible to destroy a drive is wrong, then there should be more than enough people users of Rufus with technical expertise, to disprove the claims I put forward and explain, with technical details, how the Windows Hardware Abstraction Layer, or the way Rufus accesses drives (which is through the HAL), can actually fail to protect USB Flash Drives.

So, while I do not ask you to blindly take my word for it, I will however ask you to consider that the absence of any such reports has to be corroborative that no such a major issue actually exists.

And before someone starts to rant about how it shouldn't be for others to "prove" that the application is defective, please remember that I have already explained, at length, how due to the APIs being used and the manner in which Rufus is accessing the drive (which it does in the exact same manner as it would a SATA HDD — As a matter of fact, you can tell Rufus to list and access SATA drives through a hidden cheat mode, provided you have also set these drives Removable in your BIOS, since all that cheat mode does is tell Rufus not unlist these drives), it is just as impossible for me to go with "But isn't it possible that there is some place in the code that may send the 'wrong' set of data to the drive? Why do you seem to refuse to consider that as a valid hypothesis?" as it would be to go with "But isn't still possible that the earth might be flat? Why do you seem to refuse to consider that as a valid hypothesis?". It just doesn't work that way. And this is even more difficult to accept as a valid hypothesis when:

Therefore, I can only exhaustively indicate that, while I am not denying that a small number of people appear to have experienced drive failures, that happened to occur while they were using Rufus, simply stating a "belief" that Rufus must be responsible for the failures is still not going to cut it, unless one also happens to have RMA'd their drive(s) and gotten some technical backing from the drive manufacturer (or some other specialist), to try to prove the technical aspect of the claim (and disprove my counter-claim, which, as some people don't seem to understand, I have long backed up with technical details).

lurch commented 5 years ago

We also encountered the FreeBSD Windows-bug on Etcher https://github.com/balena-io/etcher/issues/1839

pbatard commented 5 years ago

We also encountered the FreeBSD Windows-bug on Etcher balena-io/etcher#1839

No surprise there. I also replicated if with win32diskimager at the time, as well as Linux' dd and then plugging the drive in Windows.

I wish I knew what Microsoft actually fixed in Windows 8 to finally make that issue disappear. As far as I can tell, it seems to have something to do with improperly parsing GPT records...

lurch commented 5 years ago

I wish I knew what Microsoft actually fixed in Windows 8 to finally make that issue disappear. As far as I can tell, it seems to have something to do with improperly parsing GPT records...

https://forums.freebsd.org/threads/62230/ has more details: "The Memstick image is smaller than the memstick and even though it places the secondary GPT at the end of the image, that is not necessarily the end of the disk. This seems to crash Windows."

I guess Windows is assuming that if there's a valid GPT at the start of the disk, there must also be a valid GPT at the end of the disk; and it then falls over when it tries to parse a non-existent GPT from the end of the disk (which of course contains entirely random data).

pbatard commented 5 years ago

Aha. That makes a lot of sense indeed (though not the part about parsing random data = crash).

Because people have requested it in the past, I've been wondering about having Rufus fix the GPT backup record as part of the image writing for larger drives... But that's a discussion for another thread.

n30nwav commented 4 years ago

@pbatard hey buddy. I had to come here to post this after going through multiple threads and your own here (which is closed now) in regards to usb drives and sd cards being rendered write protected after using Rufus. This is what I have found. IT IS NOT ACTUALLY RUFUS's fault

not sure if the users above had the same experience (more on that below) but my sd card got write protected after I tried creating a USB using your tool. Tried multiple things to solve this (deleting partitions, recreating, using different tools, different formats, removing diskpart readonly attribute even) but it didn't work. I even tried to use the sd card on another machine and it failed.

Next time I tried to be a bit more careful and realized my corporate policies on the laptop was forcing Bitlocker to be Turned ON and encrypt the drive. And if I specifically select Don't encrypt it. The drive was supposed to stay in a read only state. In other words write protected. What's interesting is this is written in such a small font under the option which says "Don't encrypt this drive" that you can actually miss it considering it's just not going to encrypt the drive and everything else would be hunky dory. Anyway. What's interesting is even in this state. Somehow Rufus actually is able to delete the partition (which kindda make sense since Bitlocker might only be working on the FileSystem level and not on the Boot sector or partition level) and this leads the drive to be in write protected state. At this point nothing that would do on this current machine would work. And on top of that the usb/sd card will not work on any other device.

The solution to this is though is to put the USB/SD Card on another machine which is not governed by BitLocker. Run diskpart on the disk and clear the the readonly attribute by running the command "attributes disk clear readonly". Once you do that, you can use any formatting tool to format the sdcard/drive on same machine and then reuse it however you want.

Hope this may help somebody somewhere. If this post is redundant or useless. Feel free to remove it :)

pbatard commented 4 years ago

@n30nwav, I appreciate the report, which is quite interesting indeed.

I'll see if I can add a note about checking for Bitlocker in the FAQ, and try reformatting the card on a machine where Bitlocker is not active, coz this may help other folks who run into the same situation.

karan316 commented 4 years ago

I don't think it is an isolated incident.

But if these aren't, how comes that despite having tested Rufus on every single computer and every single flash drive I could get my hands on, I have never ever been able to replicate such an issue. And neither seem to have any of the experienced power users of forums such as reboot.pro, or, as I mentioned, the manufacturers of the drives Rufus is supposed to write protect.

In other words, to go with your hypothesis, we would also have to go with the hypothesis that: I and other developers and developers who are testing USB flash drives on a daily basis have been extraordinarily lucky in never ever experiencing the issue, despite the fact that, by all account if that issue is real, we would be the most affected by it.

just now confirmed the end point was Rufus.

Not really. What you did is confirm, that either one of Rufus or your flash drive has an issue, and, considering that it's only the internal flash drive controller that can decide to switch a drive to read-only and that (and I have to stress that out, because people who have no knowledge of how USB flash drive work really don't seem to understand that point) it is simply NOT possible for an external software application that only issues USB Mass Storage commands to tell the USB Flash Drive's controller to switch the drive to read-only. Instead, it's the internal logic of the controller that does that on its own, and the only factor it uses to do that is when it detects that some flash memory cells are defective (in most cases that happens when the controller itself is trying to write data to a cell, and the controller itself detects that the operation fails. Please bear in mind that there's no external entity involved in this operation besides the internal flash drive controller).

But, hey, I hear you coming up with: "But surely Rufus is sending USB commands to the flash drive. What if there's a bug and Rufus happens to send a wrong command, such as one that might tell the controller to switch to read-only?"

Unfortunately for you (and for the many people who are unaware of what USB software development entails on Windows, and think that it's therefore easy to screw up), that's not at all what Rufus does. Rufus does NOT directly send USB commands to anything. Especially we're not building any of the USB messages that are received by the flash driver controller. So, even if the application has a bug, it's never going to translate to "Oops, I wrote bad data into an USB field and inadvertently sent an USB message that tells the controller to switch to read-only, instead of simply writing data".

Instead, what happens is that we ask Windows to perform some operations, such as "read or write sector n of device XYZ", "tell me the type of device ABC, so that I can find out if it's USB, and remove it from the list of devices I present to the user if not", and that's really it.

As a matter of fact, the way Rufus is designed, it doesn't matter one bit if a disk device is USB, SATA, SCSI, NVMe, Virtual (VHD, VMWare), because there isn't a single section in the code where we construct and send actual bus commands, such as USB ones. In fact, a lot of the enumeration code of Rufus is to eliminate things like internal drives, because otherwise, Rufus would happily let you do the exact same thing to your internal SATA drives as you can do to USB drives. And, here's the kicker: I wouldn't have to change a single line of code in Rufus to make it partition and format an internal SATA drive as opposed to an USB drive, because Rufus doesn't need to care about the protocol being used when writing the drive. As a matter of fact, Rufus can already be used with external SATA drives (provided that the BIOS sees them as removable and that you use a well-hidden cheat mode), and I didn't have to do anything special in the code to achieve this, besides ensuring that less drives get eliminated during enumeration when the cheat mode is in effect.

All this to say that all we are using in Rufus are the generic Windows API commands that let you read/write a sector or a file, and that, as per Windows design, are completely bus-agnostic. Or, to put it in terms that you may also have heard, we use the Hardware Abstraction Layer of Windows, and therefore do not have the possibility of sending straight USB commands.

Alright, so that "comeback" point 1/3 covered, so let me get to "comeback" point 2: "But what if Rufus screws up the data it sends to the Hardware Abstraction Layer? Couldn't that somehow result in Windows sending bad USB commands to the flash drive?"

There again, I'm afraid that, if you know what you are talking about, you will have to dismiss the idea. The first thing I'm gonna point out is that, if this was at all possible, then surely there would be a Windows bug here, because what one expects from a Hardware Abstraction Layer API is that it will filter bad data, and prevent potentially destructive commands from being sent on the bus. In other words, if Windows was allowing anything like that, Microsoft would have heard about it from hardware manufacturers who, again, have in their interest not to have an OS that can ever send self destruct commands to hardware by mistake. Also, and this is the most important point, you have to realize that these APIs are very limited. For instance, here is the ONLY API call that is ever used in Rufus to write to an USB flash drive (either directly or indirectly, but the indirect calls always translate to WriteFile() being issued in the end). If you know a little bit about programming, you'll have to explain to me how it's possible to screw up parameters so bad that Windows will end up sending a "bad" USB command on the USB bus. If you use the wrong handle, you're not gonna be accessing the flash drive at all (unless Windows is super buggy) so we can eliminate that. And we never use Overlapped mode (which wouldn't screw us up in the first place anyway), so that leaves passing either a wrong buffer (but if you address is corrupt, you'll get a segmentation fault in the Windows application, not on the target device) or a buffer with bad data... knowing that there's nothing in the Mass Storage protocol that will make the USB bus choke on specific data content (because it's of course designed to be data agnostic, meaning that, as far as the Mass Storage USB protocol is concerned, any data you have in your buffer is just fine and there's no such thing as data that is "more valid" than other). In other words, you could write whatever random stuff you want in your buffer, it's never going to end up producing anything on the USB bus but a Mass Storage message that says "Here's some data for you, that you should write at address N". Especially, it's NEVER going to result in a non Mass Storage USB command (such as "vendor-internal: switch to read-only") being issued on the USB bus. Even if Rufus is buggy, all you'll ever get on the USB bus are regular "read sector N" or "write sector N" messages, that are the only messages Windows can produce in the way the application works, and that's it.

And with this we come nicely to "comeback" 3: "But what if you actually send garbage data for the USB controller to write, or tell it to write the data to some 'special sectors'? Surely if you do that, you may end up screwing up the drive!"

Two parts to this (NB: I'm simplifying things quite a bit here, which doesn't make the explanation any less valid, so if you want to dismiss the whole reasoning because it's not "technically accurate", you'd better be prepared to show where exactly my explanation would be disproved in the the more technically accurate picture):

  1. The idea that there exists "special sectors" on an USB flash drive, that can be accessed through standard USB commands (such as the "write sector N" we've seen above) is wrong. That's not to say that there might not exist special sectors that are being used to store the controller firmware, but, unless that controller is buggy, it should never let you access these sectors through the regular "write/read sector N" commands. Especially, the sector at address 0 should be the very first sector that the controller designates as accessible for storing user data (NB: this includes the MBR or any sector that belongs to a partition table, which, as far as the controller, Windows and Rufus are concerned ARE pure user data), and, unless it has a major bug, it should return an error whenever it receives a standard command that asks it to read a sector that is lower than 0 or higher than the total capacity the controller has reserved for user data. All this to say, even if you were to have a buggy application that somehow manages to get Windows to send a "write sector -1 with this data" command on the USB bus, the Flash drive controller will always validate that the sector it is being asked to write to belongs to the user data range, before it does anything, and will return an error if it doesn't (that is, unless your firmware is very buggy, but even then, I can tell you that Windows should have validated the sector range before the controller gets to do the same, so even if your firmware is buggy, it shouldn't actually matter). In other words, no matter how hard you try, you can not use Windows's WriteFile() (which, again, after device enumeration, is the only call that Rufus ever uses that translates into sending USB bus commands) to overwrite something like the memory area that is used by the firmware (the only exception I can see for this is if you are using a "fake" drive, but if you do, then you can't expect any application, including Windows, not to mess that drive up as soon as you write more data than to it than its actual "non-fake" capacity), let alone toggle a firmware-internal flag such as "read-only" mode.
  2. But what about partition table sectors? Surely the sectors that hold the partition table must be "special" and writing buggy data into them might be the reason why a drive could turn read-only? Unfortunately, that whole reasoning falls flat when you know that Windows (or any other OS for that matter) does not see the partition table sectors (or MBR, or VBR/PBR or whatever other set or sector you were told were "special" because if you overwrite them you could lose your data — but please don't conflate losing data with rendering a drive inoperable, since you can always reuse a drive after you lose the data it contained) as any different than any other sectors from your drive. If you can take a completely blank drive and repartition it/reformat it in Windows, or even a drive that has been secure-erased, then clearly, there's nothing that "special" about some sectors compared to other, and, even if you write garbage data on these, the worst thing you'll have to do is repartition/reformat the drive. Especially, you will NEVER end up in a situation where you actually can't recover the drive, no matter what amount of sectors were overwritten with garbage data. Therefore, it is not possible to screw up a drive simply by writing the "wrong" data to any of the sectors that Windows can access.

So, in summary, and as I have tried to establish:

  1. Contary to what you may believe, The Rufus code does not construct and send USB commands on the bus, so, even if there's a major bug, it just cannot inadvertently construct and send a "self destruct" command to a device. Instead it uses the Windows Hardware Abstraction Layer, that makes ALL disk devices (USB, SCSI, etc.) be accessed in the same abstracted manner, i.e. without any direct possibility of control over the USB bus.
  2. Even if Rufus somehow sent garbage/buggy data to the Windows Hardware Abstraction Layer (which pretty much means to WriteFile() since that's all it ever uses to write to a device), the only way it would be able to screw up a USB flash drive is if it were to write to the non-accessible firmware-internal flash drive memory, by attempting to write to a sector that is outside of the regular range for the device. Unfortunately, the only way that could ever happen is if both Windows and the flash drive controller are buggy, because both these entities are supposed to validate the range being accessed, and reject the request if the range is invalid. That's simply not realistic (but if you believe it is, then by all means you should complain to your hardware manufacturer first, because you have just acknowledged that they have done a very crappy job).
  3. And finally, with the knowledge that, even if it was buggy, the only thing Rufus can only ever realistically access are the actual user data sectors reported by the device, and that none of these sectors (partition table, etc.) are any special, and that any competent OS will always be able to rewrite them, no matter the garbage data they contain, we have pretty much demonstrated that it is simply not logically possible for an application like Rufus to render a USB flash drive read-only or make it fail, UNLESS THAT FLASH DRIVE ALREADY HAD A HARDWARE ISSUE IN THE FIRST PLACE (in which case, writing a large amount of data, which is what Rufus typically does, will produce the same issue, regardless of the application being used).

How's that for a deep analysis?

Of course, you're welcome to ramp up your expertise on Windows APIs, Mass Storage command and USB, as well as spend some time with the Rufus source, to tell me why you (and others), still don't think what I went great length to explain above is unsatisfactory and why, surely, because you happened to experience a couple of drive failures while using Rufus, whereas millions of other Rufus users don't seem to ever have run into such such issues (and flash drive manufacturers also seem to be strangely quiet about the "destroyer of flash drives" that Rufus allegedly is), it must mean that Rufus is the culprit rather than your hardware...

PS: I don't think the majority of the people know about Github in the first place

You do realize that you're trying hard to justify why, if the problem is so widespread, the hundreds or thousands of people who have to be affected by it, out of the 3 millions that download Rufus every month, would choose not to report it.

First of all, means of contacting me by e-mail are clearly available both on the Rufus homepage and in the About dialog. And (but you'll have to trust me on this), the amount of "Rufus destroyed my flash drive" e-mail reports I get is about the same as I see from other means. Also, people do use stackoverflow, reddit, and other forums such as mydigitallife, which I monitor, and likewise, you'll be hard pressed to see the tens or hundreds of reports one would expect to see there if Rufus had such a flaw as the one you describe.

Your justification as to why so few people would report such an issue, if it was a real one, clearly does not add up, which, I'm afraid, can only mean one thing: the hardware issues people experience while using Rufus are coincidental.

Oh, and for the record, there does happen to exist buggy firmwares out there, that seem to make some brand/models of USB flash drive fail more frequently than others. For instance, as per the List of incompatible hardware from the FAQ, some PNY drives have a dodgy firmware (which the manufacturer acknowledged), and I suspect that, if you have a couple such drives, and use them with Rufus, you will find that they consistently fail, even though this has nothing to do with Rufus (the issue should happen if you write a lot of data, regardless of the application being used).

Finally, for anyone still tempted to try to push this "Rufus can destroy flash drive" story again, I'll say that any argument you are trying to make will have a lot more weight if you are able to report something like: "Hey, I bought a couple of drives from a reputable vendor (e.g. SanDisk, Adata, etc.), but when I used them with Rufus, these drives switched to read-only. So I RMA'd the drives to the vendor, describing what I did, to ask them if they could replicate the issue. They have now come back to me and confirmed that Rufus is the source of the problem because [insert explanation from people who actually have some expertise on the matter here]...".

Again, if the "problem" was as widespread as you say, and I as I pointed out before, there's no way flash drive manufacturers would choose to ignore it, and you should be able to get strong evidence about Rufus being the cause of the issue from the manufacturers themselves or power users with some USB expertise (e.g. people who know how to run an USB bus trace). Yet, despite the isolated but usually very voiceful complaints I have seen here and there about Rufus damaging flash drives, and despite Rufus having been quite popular for years, I'm still waiting for such reports...

Even I am facing the exact same problem. The diskpart solution didn't work because the readonly state is already set to NO. The current read only state is YES. However when I use the pendrive on Linux it works just fine. And this is the second time happening with two different pen drives. Don't know what's going on.

pbatard commented 4 years ago

However when I use the pendrive on Linux it works just fine.

Do you have BitLocker enabled on your machine, or some other security policy?

Obviously, you are not experiencing a hardware failure, if your drive works just fine on Linux, and as @n30nwav pointed above, it appears that, on some platforms that have BitLocker enabled, the default BitLocker policy may enforce a drive to be seen by the system as read-only as soon as you repartition or reformat a drive that is not set to use BitLocker (which would be the default for Rufus).

At this stage, I'm still waiting to hear for more reports to validate this, so that I can potentially try to detect the precise setting that decides this policy and see if I can try to warn users about it in Rufus.

karan316 commented 4 years ago

However when I use the pendrive on Linux it works just fine.

Do you have BitLocker enabled on your machine, or some other security policy?

Obviously, you are not experiencing a hardware failure, if your drive works just fine on Linux, and as @n30nwav pointed above, it appears that, on some platforms that have BitLocker enabled, the default BitLocker policy may enforce a drive to be seen by the system as read-only as soon as you repartition or reformat a drive that is not set to use BitLocker (which would be the default for Rufus).

At this stage, I'm still waiting to hear for more reports to validate this, so that I can potentially try to detect the precise setting that decides this policy and see if I can try to warn users about it in Rufus.

Now, even on Ubuntu it says the USB is read-only and cannot format it. I've tried all possible ways on the internet to fix it but to no avail. I also tried to make it non bootable (since it showed the pendrive was bootable) using Rufus which didn't work either because of the read only state.

karan316 commented 4 years ago

Also if the BitLocker policy is enabled, how do I get around it? I've created bootable pen drives from the same laptop previously and didn't face any issues. Yesterday I tried to create a Live USB of Manjaro and Rufus suddenly showed that the process had FAILED. I am facing this read-only state problem since then.

pbatard commented 4 years ago

Also if the BitLocker policy is enabled, how do I get around it?

I'm afraid I don't have that information. I'm just reporting what was mentioned above.

But really, if your drive has been seeing what appears to be an intermittent failure, and then failed altogether, then it all points out to your flash memory being defective, which is the most common issue USB Flash Drives face, i.e. a pure hardware failure that wasn't induced by any software operation.

A lot of people seem to be of the idea that: "If I was using Rufus when writing the drive, and the drive failed, then it must mean that Rufus has a bug", and yet, if the exact same thing happens (as it also commonly does) when using Windows utilities only, they automatically think "hardware failure" and let Microsoft off the hook, on account that, surely an entity like Microsoft would never ever introduce major bugs in Windows that can lead to complete destruction of the data you are explicitly trying to protect. But I'm afraid that is not how it goes. Hardware failures are common, especially for Flash Drives, because USB Flash Drives aren't built for as much long term reliability as other media, and you can't just point the finger at one specific application on account that you've seen a couple of pen drives fail while you were using it.

All I am seeing from your report above points to a pure hardware failure and not anything that Rufus (or BitLocker for that matter) seems to have much of anything to do with. So your best course of action at this stage is to RMA your flash drive if it's still under warranty, and purchase a new one, as it's very unlikely you'll be able to salvage it.