[MT12] “No SD Card” issue with nightly build

chofchop commented 6 months ago

Is there an existing issue for this problem?

[X] I have searched the existing issues

What part of EdgeTX is the focus of this bug?

Transmitter firmware

Current Behavior

When I update my MT12 to nightly build firmware, I encounter "No SD Card" for certain brands of SD cards. With MT12's stock firmware, all my SD cards work fine.

Expected Behavior

All of the SD cards I own can be used with Windows 10 and Android without any errors. These SD cards should also be usable with EdgeTX.

Steps To Reproduce

1.I can use all my SD cards with MT12 stock firmware. 2024-01-10 17 31 03 The 512MB on the left is the one that comes with MT12. These four cards can be used even if the firmware is updated.

2.The following SD cards will become unusable due to a "No SD Card" error when updated to nightly build. 2024-01-10 17 30 47 These five SD cards can be used without any problem with MT12 stock firmware (The sandisk card may be fake).

Version

Nightly (Please give date/commit below)

Transmitter

RadioMaster MT12

Operating System (OS)

No response

OS Version

No response

Anything else?

The four SD cards that can be used for nightly builds will not start unless the power button is held down for more than 1 second, even if "Pwr On delay" is set to 0 seconds.

The five SD cards that cannot be used in nightly builds will boot in less than 0.5 seconds by pressing the power button.

3djc commented 6 months ago

cannot reproduce the issue
if "Pwr On delay" is set to 0 seconds, it does not mean that a short push will start the radio, you still have to push until you get haptic feedback. That settings means EdgetTX doesn't add any extra delay, but you still have the hardware one.

chofchop commented 6 months ago

if "Pwr On delay" is set to 0 seconds, it does not mean that a short push will start the radio, you still have to push until you get haptic feedback. That settings means EdgetTX doesn't add any extra delay, but you still have the hardware one.

I know that. What I wanted to say is that the time it takes to get haptic feedback varies depending on the SD card. I just wrote this because I thought it would help solve the "No SD Card" problem.

pfeerick commented 6 months ago

If they are fake, that probably means they have an even crappier controller than is usual for cheap SD cards, and it's unlikely they will ever work with this or many (if any) of the EdgeTX radios... there is only so much we can do for compatibility. If you are running a relatively recent nightly (from within the last three weeks)... then it will be the most compatible it can be for now... we adjusted the init retry time to make some other slow to init cards work in https://github.com/EdgeTX/edgetx/pull/4450 but that is probably about the best it can be.

@3djc what are the implications if we increase that time further ? Just wondering if it's worth doing a firmware build with say double that time just to see if it makes any difference for those cards.

chofchop commented 6 months ago

I can't deny that my SD card may be of poor quality, but that doesn't explain the "No SD Card" issue.

All my SD cards work with MT12 (stock firmware) and X9Lite (EdgeTX2.8.1). It is reasonable to assume that some recent change has made it no longer recognize certain brands of SD cards. Is it wrong?

As I said before, my Sandisk and QUIO cards boot faster than other cards. I think #4450 is compatible with slower SD cards, but isn't it possible that this has a negative effect on faster cards?

pfeerick commented 6 months ago

Unfortunately, it does... the MT12 stock firmware is actually a build of 2.10 from before some code that has some impact on the timings was introduced, which should be mostly mitigated by #4450. Those "faster" (in reality probably even slower) Sandisk/QUIO cards are probably not responding soon enough, hence why the failure message is shown so quickly (giving the false perception of a quicker boot). #4450 is increasing the time afforded for init retries... i.e. to give slower cards even more time to respond. I would expect that quicker cards would respond sooner, so there would be no need to wait for them in the first place.

btw, What exactly do you mean by this ...

The five SD cards that cannot be used in nightly builds will boot in less than 0.5 seconds by pressing the power button.

... are you saying that those cards will show the "No SD card" error in like 0.5s from pressing the power button? As that is ... interesting... i.e. if you pull the SD card out, it should take like 3 seconds for that message to show... 🤔

chofchop commented 6 months ago

... are you saying that those cards will show the "No SD card" error in like 0.5s from pressing the power button? As that is ... interesting... i.e. if you pull the SD card out, it should take like 3 seconds for that message to show... 🤔

No, it's not. This happens with MT12 stock firmware.

For the four SD cards that can be used for nightly builds, It takes more than 1 second to get haptics after pressing the power button.

Sandisk, QUIO card gets haptics and starts EdgeTX within 0.5 seconds after pressing the power button.

I'll make a video if you need it.

pfeerick commented 6 months ago

Sorry, I should have been clearer... I meant to ask what are you defining as "boot"... so it's the haptic feedback?

In that case, there seems to be something else going on here, as I think until the SD card is read, the default pwrOnSpeed delay is two seconds (or 1 second for some specific builds)... with a missing/unmounted SD card honouring that delay, and then the error message with haptic at pretty much the same time as the error message is shown. So haptic less than one second from pressing the button would to me suggest that it actually is reading the card, but by the time later in startup that it does the if (!sdMounted()) check as part of the decision to show the error, for some reason the card isn't responding.

A video certainly wouldn't hurt, as there is no confusion at all as to exactly what is happening. Maybe the stock firmware with one good, one bad card. And then same again with the nightly you've been using? Presumably a recent one, as you never mentioned the hash or date?

chofchop commented 6 months ago

Sorry, I think my explanation was unclear and caused some misunderstanding.

I'm going to make a video, but I'm not used to it so I think it will take some time.

3djc commented 6 months ago

@3djc what are the implications if we increase that time further ? Just wondering if it's worth doing a firmware build with say double that time just to see if it makes any difference for those cards.

The main issue are EM reboots. For those cards that are super slow to turn on, SD availability is on the critical EM recovery path. That delay could be extended, but it would mean EM recovery would be longer. Given the price of an SD card and a model, wisest to change the SD to a decent one than the risk of loosing your model imho.

https://github.com/EdgeTX/edgetx/pull/4450 will not affect faster card, since it is a max loop value. Good cards get out of the ready wait loop much faster than current or previous timeout value anyway, so for those nothing has changed.

The SD hal rewrite (#3957) is a complete rewrite of the SD accesses, but if anything, a more standard one. Some card that where not ok with older builds (like mt12 factory firmware) can become ok, some other that where ok won't anymore.

I have here 4 genuine Sandisk SD card, all work fine

Capture d'écran 2024-01-12 084520

pfeerick commented 6 months ago

The main issue are EM reboots.

That was my main fear... and I fully agree... the fact this is not working is probably a good thing... like refusing to work with a cheap "fake" USB or SD card that is not really the size it says it is... and is guaranteed to start eating your data when you use more than the real size it is. 😖 🤮

I do get that it is also frustrating when you have a microSD and it doesn't work... I been there... but I think it's better to keep your life, limbs and property safe... and fix the real problem. Can't wait for more transmitters like the T20 with no SD cards! :)

chofchop commented 6 months ago

I made a video. https://www.dropbox.com/scl/fi/ufvzdf8ijx8ntf2eezzsa/VID_20240112_192953.mp4?rlkey=wkk3nufn2uza5m53m9azn3k96&dl=0

1.First up is the MT12 stock firmware and stock SD card. Please turn up the volume so you can hear the haptics.

2.Next, replace the SD card to Sandisk. You can clearly see that the startup time has become faster.

3.When updating the firmware to nightly build, a "No SD Card" error appears.

4.It will start when I put it back into the stock SD card.

chofchop commented 6 months ago

Given the price of an SD card and a model, wisest to change the SD to a decent one than the risk of loosing your model imho.

I also agree with that opinion. However, you seem to be blaming this problem on a poor quality SD card, but I don't believe that to be the case.

My SD cards are completely divided into those that can be used for nightly builds and those that cannot be used depending on the brand. Perhaps the specifications of the two are different somehow. But that doesn't mean the quality of either one is extremely bad.

By the way, all of my SD cards are tested for full capacity read/write and transfer speed to confirm that they have sufficient performance, so they are different from so-called inferior products such as those with falsified capacities.

chofchop commented 6 months ago

I do get that it is also frustrating when you have a microSD and it doesn't work... I been there... but I think it's better to keep your life, limbs and property safe... and fix the real problem. Can't wait for more transmitters like the T20 with no SD cards! :)

I'm not complaining about not being able to use my SD card. By pointing out the problems and getting them fixed, I hope that EdgeTX will become better.

chofchop commented 6 months ago

Is there a way to try a specific older build to see at what point this issue occurred?

3djc commented 6 months ago

Given the price of an SD card and a model, wisest to change the SD to a decent one than the risk of loosing your model imho.

I also agree with that opinion. However, you seem to be blaming this problem on a poor quality SD card, but I don't believe that to be the case.

My SD cards are completely divided into those that can be used for nightly builds and those that cannot be used depending on the brand. Perhaps the specifications of the two are different somehow. But that doesn't mean the quality of either one is extremely bad.

By the way, all of my SD cards are tested for full capacity read/write and transfer speed to confirm that they have sufficient performance, so they are different from so-called inferior products such as those with falsified capacities.

The issue here is 'turn on' time: the time it takes from power applied to the card responding ready for use. We know exactly when this issue started (#3957). Timing have been relaxed in #4450/ but at this time, we are not considering extending it further

chofchop commented 6 months ago

The issue here is 'turn on' time: the time it takes from power applied to the card responding ready for use. We know exactly when this issue started (#3957). Timing have been relaxed in #4450/ but at this time, we are not considering extending it further

I never said anything about extending that time. As I showed in the video, the Sandisk card, which has a "No SD card" in the nightly build, is much faster to boot. Let's forget about #4450 for a moment. I think the problem is occurring elsewhere.

chofchop commented 6 months ago

We know exactly when this issue started

Since you haven't reproduced the problem I'm experiencing, there's no way you know when the problem started occurring.

richardclli commented 6 months ago

Seems all B/W radios used SPI based SDCard implementations, and currently when I walk through DMA implementation for SPI, I suspect that the dma of SD card access via SPI may have some problems.

This ls related to thhis PR https://github.com/EdgeTX/edgetx/pull/3957

The problem I suspect is the DMA setup use FIFO with full buffer threshold and single burst. This can make the buffer overflow if the SD card is responding fast. I changed the settings in a related branch and see if it works or not.

@chofchop Could you please try to build the firmware from the following branch and see if this works for you?

https://github.com/EdgeTX/edgetx/tree/spi-flash-dma

chofchop commented 6 months ago

I'm trying it now. Please wait a little.

chofchop commented 6 months ago

I did the following:

I opened the spi-flash-dma branch on Gidpod. https://gitpod.io/#https://github.com/EdgeTX/edgetx/tree/spi-flash-dma
I executed the following three lines one by one. cmake -Wno-dev -DPCB=X7 -DPCBREV=MT12 -DDEFAULT_MODE=1 -DGVARS=YES -DFONT=SQT5 -DPPM_UNIT=US -DHELI=NO -DLUA_MIXER=YES -DCMAKE_BUILD_TYPE=Release ../ make arm-none-eabi-configure make -C arm-none-eabi -jnproc firmware
I downloaded Gitpod's build/arm-none-eabi/firmware.bin, transferred it to MT12's FIRMWARE directory, and updated the firmware from the bootloader.

Unfortunately, the "No SD card" error cannot be resolved.

richardclli commented 6 months ago

I updated the branch by using the direct DMA mode, see if this helps.

richardclli commented 6 months ago

Anybody knows if the old driver uses DMA or not?

chofchop commented 6 months ago

I tried build it again with Gitpod, but the "No SD card" error still persists. This is my feeling and there is no basis for it, but rather than not recognizing the SD card, it is more like the SD card is not present.

pfeerick commented 6 months ago

Just FYI, there is no need to build the PR firmware yourself unless you want different build options... every PR commit is built automatically... just go to the Checks tab on that PR, then "Run tests and package firmware", and under Artifacts heading down the bottom will be edgetx-firmware-merge which is the zip file with the firmwares for all the supported radios for that build.

3djc commented 6 months ago

@chofchop you probably want to ensure you do a DFU flash using Buddy, not just a firmware flash using bootloader

chofchop commented 6 months ago

@chofchop you probably want to ensure you do a DFU flash using Buddy, not just a firmware flash using bootloader

I'll try. Please give me a time.

chofchop commented 6 months ago

I tried to flash the spi-flash-dma branch firmware with EdgeTX Buddy, but there was no change so far.

chofchop commented 6 months ago

If it helps the analysis, I can send you my SD card. You need it?

chofchop commented 6 months ago

Just FYI, there is no need to build the PR firmware yourself unless you want different build options... every PR commit is built automatically... just go to the Checks tab on that PR, then "Run tests and package firmware", and under Artifacts heading down the bottom will be edgetx-firmware-merge which is the zip file with the firmwares for all the supported radios for that build.

Excuse me, please tell me.

Is the firmware that can be obtained using this method the firmware at the time the PR was merged (no subsequent PRs were merged)?
How can I obtain the latest firmware that was just before that PR was merged, for example, PR #3957 was not merged?

pfeerick commented 6 months ago

It is build of the latest commit of the branch the PR is based on. So until it is merged, any other PRs that might have been merged between the initial commit and the current state of the branch do not exist.
If it is older than the last two weeks, your only option is to build it yourself... You would look at the commit hash that relates to it being merged into main, and then want the commit immediately prior to it.

i.e. you can see it is https://github.com/EdgeTX/edgetx/commit/87df8a4f2cc741c99d4c52e9f99c4af5ca2c9b2d from on Sep 23, 2023. Looking at the day before and after, this is the immediately prior commit: 870fe7eca3e96f97c6a744087a109fa9556eb080

richardclli commented 6 months ago

I do not have a MT12, so I cannot help any further after these 2 trials.

chofchop commented 6 months ago

I confirmed that the same problem occurs with X9Lite. I also confirmed that it starts from #3957 (87df8a4). This does not occur on 870fe7e.

3djc commented 6 months ago

Clearly, occurs on all radio, is a direct consequence of #3957 as I said earlier (https://github.com/EdgeTX/edgetx/issues/4530#issuecomment-1889265360)

chofchop commented 6 months ago

If this problem is unique to me (my SD card), I don't think there is any need to pursue it further. In fact, I think it would be better for EdgeTX if that were the case.

However, if #3957 has a potential problem, I think it is necessary to resolve it thoroughly.

I can't decide which situation I'm in right now.

richardclli commented 6 months ago

@raphaelcoeffic Obviously, it is your turn now.

pfeerick commented 6 months ago

Thanks for the video you shared back at https://github.com/EdgeTX/edgetx/issues/4530#issuecomment-1888887331. I can clearly see the increase in boot speed you referring to before, and that you were using the haptic as the reference.

I've tried several different cards now, including some claimed 128MB Sandisk cards I got cheap for datalogging, some 512MB that came with some other cheaper transmitters, and some genuine Sandisk Ultra (32GB and 16GB). Unfortunately I've only been able to come across two cards (the 128MB cards, and one of two 512MB no name SD cards - the second one was fine) that would not work... but when I checked them against the TX12MKII they didn't work there for 2.9.2 or latest nightly, so I would say they were simply not compatible at all. I've not checked against https://github.com/EdgeTX/edgetx/pull/4478 to see if that makes any difference... will try that tomorrow.

There is no denying that https://github.com/EdgeTX/edgetx/pull/3957 introduced some issues regarding compatibility. The question is going to be one of

is we introduced a bug?
is there a bug in the code it was based on (because that has never happened before 🙄🤭)?
is it a borderline compatibility issue?

From my side, I think we'll have to see what Raphael thinks of the issue. Getting one of those cards off of you may indeed be needed to help resolve it if we can't dig something out from our own stashes. Since it's known to be problematic, and there is no guarantee if we ordered some OUIO cards we'd get ones that don't work. Just have to see if there is anything more we can do "in the dark" before taking you up on that offer ;)

fiechr commented 6 months ago

Is it possible to get some more data from the cards not working? Like, for example, partition table, partition alignment, size, file system type, cluster size?

I also have a SanDisk 32 GB Ultra and it works fine in the MT12 with all the latest builds I tested.

sandisk32gbultra

$ sudo sfdisk -l /dev/mmcblk0
Disk /dev/mmcblk0: 29.73 GiB, 31927042048 bytes, 62357504 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x994645f8

Device         Boot Start      End  Sectors  Size Id Type
/dev/mmcblk0p1       8192 62357503 62349312 29.7G  c W95 FAT32 (LBA)

$ sudo dosfsck -v /dev/mmcblk0p1
fsck.fat 4.2 (2021-01-31)
Checking we can access the last sector of the filesystem
Boot sector contents:
System ID "MSWIN4.1"
Media byte 0xf8 (hard disk)
       512 bytes per logical sector
     32768 bytes per cluster
      1164 reserved sectors
First FAT starts at byte 595968 (sector 1164)
         2 FATs, 32 bit entries
   3896320 bytes per FAT (= 7610 sectors)
Root directory start at cluster 2 (arbitrary size)
Data area starts at byte 8388608 (sector 16384)
    973952 data clusters (31914459136 bytes)
63 sectors/track, 255 heads
      8192 hidden sectors
  62349312 sectors total
Checking for unused clusters.
Checking free cluster summary.
/dev/mmcblk0p1: 2413 files, 3655/973952 clusters

$ cat /sys/block/mmcblk0/device/cid
035344534433324785fc000001016300

According to the CID, my card was manufactured in 2016 and the product revision is 8.5 (I removed the serial number part): https://archive.goughlui.com/static/cidecode.htm (CID decoder)

chofchop commented 6 months ago

Is it possible to get some more data from the cards not working? Like, for example, partition table, partition alignment, size, file system type, cluster size?

I'm not familiar with Linux. How can I do this on Windows 10? By the way, my Windows 10 has an MSYS2 environment for build EdgeTX.

fiechr commented 6 months ago

I'm not familiar with Linux. How can I do this on Windows 10? By the way, my Windows 10 has an MSYS2 environment for build EdgeTX.

diskpart should be able to show you the partition table and the alignment value. It's a a bit far fetched, I have to admit. But I was recently experimenting with the official SD formatter tool and found that it aligned the partition at sector 8192 (4 MB) instead of just 1 MB or 2 MB which is the usual value the OS tools use. But I also don't want to waste your time by chasing ghosts. What maybe could be an idea, is to try format the card with the tool from the SD Association just to eliminate the partitions and formatting as a possible source of issues. If it doesn't work then either it's probably a hardware thing or not the cards "fault".

Windows: https://www.sdcard.org/downloads/formatter/sd-memory-card-formatter-for-windows-download/ MacOS: https://www.sdcard.org/downloads/formatter/sd-memory-card-formatter-for-mac-download/ Linux: https://www.sdcard.org/downloads/sd-memory-card-formatter-for-linux/

chofchop commented 6 months ago

...try format the card with the tool from the SD Association just to eliminate the partitions...

I was already using SD Card Formatter 5.0.2. I did a full format (not a quick format) with SD Card Formatter for the SD card that showed "No SD card" in the nightly build, but there was no improvement.

I downloaded a new SD Card Formatter from your link and reinstalled it, but no improvement was seen.

greenbigfrog commented 1 month ago

I experienced this as well migrating from a 2020 openTX version to edgeTX v02.10 on my X9D. The original SD card (2GB Toshiba SD-C02G) did not work after flashing to edgeTX. It was formatted as FAT16 but even after switching it to FAT32 it still does not get detected. After a bunch of googling and wondering what was going on, I found this issue and gave it a shot: I switched to a other SD Card and it worked just fine. (I did not flash back to openTX to check if it still works there or try other edgeTX versions.)

msun12000 commented 1 month ago

So what was the solution?

greenbigfrog commented 1 month ago

I switched to a other SD Card and it worked just fine.

I just realized though, this would've better fit in #5015 (or well, the issues are duplicate of each other)

chofchop commented 1 month ago

This issue has not been resolved. The developer says, "Please replace with a well-known brand SD card." It is a confirmed fact that this problem occurred from #3957.

As I understand, #3957 changes the SD card driver from STM32 HAL to LL. Perhaps there is a problem with ST Micro's LL library itself. If not, there is a problem with #3957 itself, but unfortunately I don't have the skills to confirm that.

EdgeTX / edgetx