Cross-link

[CecilFF4]

For many months, maybe longer, I've periodically had problems on my consoles' SD cards where the files and folders would accrue errors, predominantly cross-linked entries (see the "repair" pic). This happens on all of my consoles, on multiple SD cards. I use three file-managers (microSD manager, FBI, godmode9) and I could not pin down which was causing the problems until today.

I was doing some simple rename operations, in this case on my ID0 folder. After finishing the rename and saying "no" to relocking permissions, I saw that which is shown in the "cross-link" pic.

Again, this has been going on for a long time, across many versions of gm9. And it happens on both New and Old consoles, on small and large SDs. But since other people do not seem to be experiencing this problem, that leads me to only one plausible cause: perhaps I perform actions within gm9 too fast. The reason I think this is the problem is because my Nintendo 3DS folder was just fine before my recent rename, then I renamed, put in the key combo, then rapidly pressed "B" (to say "no" to relocking the permissions). Immediately after pressing "B", I saw what you see in the pic.

This is the first time I saw the cross-linked duplicate entries directly appear; usually the cross-linking occurs and I don't notice until I start seeing my console do funny things after boot (i.e. save file corruptions, apps crashing). Renaming files is my most common action in gm9, though I copy/paste/move a lot as well.

My SD cards are always formatted as FAT32 with 32k clusters (64k clusters on my 128GB card due to the GBA display glitch). If you require more information or further testing from me, please let me know.

[d0k3]

Alright... that's weird. The write permission thing can't have anything to do with it, cause the write permission system is not a SD or even FAT function. It's purely software, and it doesn't change anything in your SD card.

So, I guess I will still need more info. The faulty entries, are those entries you renamed in GM9? Can you say for absolutely, 100% sure that your 0:/Nintendo 3DS folder was flawless before the rename? Very often, problems in a filesystem manifest only some time after they occured. Also, are you 100% sure that problem cannot be caused by only one of your consoles (maybe one with a faulty SD unit)?

[CecilFF4]

Sorry for not being clear, but I didn't mean to imply that the write permission input had anything to do with the problem. I was simply thinking that my input of a command quickly, regardless of that command, prevented the write buffer from completely emptying... or something along those lines. I'm just guessing at this point.

As for the faulty entries, I only renamed the primary ID0 folder; the other three were not renamed. This situation was unique because those 4 folders were fine before the rename (yes, 100% sure, because I'm always monitoring for errors because of this problem happening multiple times), then immediately after I declined relocking write permissions, all 4 folders triplicated themselves, and I saw it as it happened.

There were instances in the past where I strongly suspected GM9 was causing the cross-link problems, but waited for more info before bringing it to your attention. One such instance was me using a script, which had rename commands only, and the script, which always succeeded previously, failed, and a subsequent scan of the SD revealed cross-links. It's hard for me to imagine that one of the other file managers I use would be to blame because one is the official Nintendo app (microSD Management) and the other was FBI, which everyone and their grandmother uses, but by its nature is an app where commands can't be input very fast.

I keep constant backups and scan for errors every time one of my consoles' SDs is in my computer. 90% of the time I'll have an error when I notice something "off" about the operation of the console. I've had 6 consoles over the lifetime of the 3DS, one of which (O3DS) was never hacked and never had problems until the screen stopped calibrating. One of my O3DSXL units was hacked, but it broke due to a microfuse issue a long time ago and I don't remember if I had cross-link errors (this was pre-GM9, btw). My other 4 consoles (O3DSXL, O2DS x2, N3DSXL) all displayed cross-link errors at one point or another, and all had/have GM9. My second O3DSXL did have problems with the SD reader eventually (instant power-off if I pressed on the area where the SD reader was located), but while that console was working, I still got occasional cross-links.

Also, I always update GM9 when you release an update. All my consoles' GM9s are on 1.6.3 atm.

EDIT (I finished some tests after I wrote the above info): Since posting about this problem yesterday, I opted to do a test where I put an error-free SD card in a console (O2DS ... the cross-link issue I originally reported on occurred on my N3DSXL). During this test I never loaded any app other than GM9. I performed random operations (rename, TMD verify, CIA creation, partition info scan, copy and paste/move). I never seemed to encounter any problems on initial load and for a short time thereafter. However, after performing a few operations, I did eventually hit an error where a CIA creation of a 600MB file was slated to take over 30 minutes to complete (normally it's much faster than that). In addition, the title info for the selected app's TMD failed to display (it was just MK7, btw). A restart of the app allowed the title info to show, but the CIA creation still failed (by virtue of it taking too long). This is one of the types of errors I've seen in the past. In my computer, the SD revealed lost chains, another common error, though no cross-links this time. Again, I only used GM9, so I'm sure we can rule that it is the cause of errors if the app is used for long enough.

[d0k3]

Okay, understood. So, apparently it's not the SD cards or the consoles that is at fault here. It may be something in how you are using GodMode9. I fear there won't be an easy, quick fix.

Can you try the most recent nightly? That one has an updated SD driver which may or may not fix stuff. Also, it has an updated FatFS library. Won't be easy to confirm this, though.

[d0k3]

Okay, and more... there must be something common to all those times that problem happened for you. Is there anything you do frequently in GM9 someone else may do not? What are the brands of your SD cards (do you stick to one?)?

Try the nightly once you can, maybe that solves it already.

[CecilFF4]

I won't be able to test until after the weekend, I think. But to answer your other questions:

My most frequent actions in GM9 are renames, followed by copy/paste/move, and app and dbs mounting. I do more, but those are my top things.

My SD brands are all over the place. Kingston, Transcend, Team, and more. All of different sizes, but all partitioned to FAT32.

[CecilFF4]

Using a verified no-error SD, I performed all the usual operations in GM9 that caused issues before. I did these operations for about 15 minutes, far longer than I'm ever casually in GM9 and longer than the time it took to get an error in the past. I did not receive any errors on my scans yet. I'll continue to use it normally and let you know if anything sketchy happens. If I don't respond in X amount of time with some fresh errors, feel free to close this comment when you see fit. Thanks for an amazing piece of software and thank you for updating it continuously!

[CecilFF4]

Sorry for the false hope. Got a whole bunch of cross-links. Did not use any other file manager software. Booted the OS a few times and didn't run any apps other than System Settings and eShop. Only did renaming before the error occurred. A fresh scan was run after the previous tests and before my normal usage resumed.

Was running GodMode9-v1.6.3-97-g57266a90-20180614134511.

[d0k3]

Okay, dammit. Are there any directories where this does happen often? Ie. the files and directories that you typically rename, where are those located?

I guess we should try with some testbuild now. I already do have a few ideas.

Also, does anyone else have that problem?

[CecilFF4]

I'll check later if my scanner has been keeping logs, but I'm pretty sure that most errors happen to child folders within the Nintendo 3DS folder. In addition, last night's glitch involved duplication of my non-ID0 folders and complete erasure of my primary ID0. After repairing the cross-links and non-valid folder entries, my ID0 had to be restored from backup (about 60GB).

It's not terribly likely anyone else would have this problem as most people don't use the multiple profiles method i outlined in my guide, which primarily involves renaming child folders in the Nintendo 3DS folder.

The renames are instructed to happen through the 0 drive, btw, and not the A drive.

[d0k3]

Okay, still no breakthrough, but keeping the discussion alive. Eventually we'll get to the bottom of this.

This test build will detect any problems right as they happen: https://f.secretalgorithm.com/mSWpU/godmode9.firm It will do renames a little slower (I think not noticable), because it will doublecheck after each rename operation and it will show an error message in case something went wrong. Can you use this from now on and continue to keep an open eye for any problems?

Also, an interesting read: http://elm-chan.org/fsw/ff/doc/appnote.html#critical As you see, the rename really is a pretty critical operation, and a failure (hardware, driver, software?) can result in corruption.

[CecilFF4]

I really appreciate you looking into this, even if it doesn't affect many people yet (or it does and they don't realize gm9 is causing an issue or that they have cross-links). I will try out the latest firm for a while. I'll try to do some tests later tonight, including focusing on renames.

[CecilFF4]

Tests are good so far. I'd hold off celebrating for a few days, but I really tried breaking it tonight and it held up perfectly.

[d0k3]

Well, I actually didn't fix anything on purpose, I only added some testing code. Is it possible that, after you did those renames earlier, you quickly took out the SD card or turned off the console (so that, possibly, some operation in progress could not be completed)?

[CecilFF4]

No, I didn't quickly take out the SD or turn off the console immediately upon completion of a rename. However, I use scripts a lot, and I always disable previews, so those scripts run very fast. And the ID0 rename is the first of many renames in my scripts. It seems possible that the next command was just instructed to occur too quickly after the previous and your delay that you coded in was exactly what was needed. I'll keep messing around with it. If it doesn't cross-link by next week, I'm tempted to call it good.

[d0k3]

I guess you's still testing, and it may not be a bad idea to still leave this open for a few days. Just a quick headsup that I now included that fix in the nightly.

[CecilFF4]

I've tried breaking it a couple other times. Mostly I've just been using it as I would normally. Seems fine. Again, thanks for an awesome app.

[d0k3]

@CecilFF4 - one week has passed with no further error reports. So, I suggest we close this issue once you say it's okay. We can also reopen at a later point if anything comes up.

[CecilFF4]

I've been using all three of my consoles under normal conditions, all with the commit of gm9 you gave to me. I haven't had any issues yet.

[CecilFF4]

Sorry to reopen this, but I just had gm9 cross-link my ID0 (and other children of Nintendo 3DS) again. It happened on a rename immediately after I had the OS auto-create a new ID0 (I deleted the old one in gm9, booted up, new one created, back to gm9 for a rename). Still using v1.6.3-49-gff86444e.

[d0k3]

Alright - and you had no error message? That build you got there should check the rename immediately and show an error prompt in case something got wrong.

Maybe it happened when you created that folder?

[CecilFF4]

It displayed my personal error message that I set in my script. I didn't see anything else.

And I didn't create the ID0. The OS did (the "Creating Home Menu Management Information" message). I doubt the OS messed anything up else this would be a much bigger and more widely known problem.

[d0k3]

So you at least got an error message? That means the checks are working as they should. Could you continue testing this with the most recent nightly build?

Also, were there any other operations you did before the rename that threw the error message and caused the problem? What operations did you do afterwards?

[CecilFF4]

I think you misunderstood me. It only showed me my personal error message that I set in my script. It has showed that before when a cross-link interrupts the script. It did not show your message. So either the checks are not working as they should or something interrupted the checks themselves. There were no other operations I performed in GM9 prior to the renaming script. After the error, I took out the SD and repaired it.

[d0k3]

I didn't even add a custom message. Assuming the checks work, though. I guess the only difference then is that the error was detected right when it happened and not on the next operation.

I meant the renaming script. Where in the script did it happen, what operations came before, what operations came after?

[CecilFF4]

Well, this wasn't a new event, my error showing when a cross-link occurs during my script; it's happened before. Maybe about half the time. So it's a little weird and random.

Anyway, here's my script (renamed as txt so github allows it). It does the ID0 rename first because that's where the errors always occur. One of three things happens on that step.

1. It completes the rename fine.
2. The script takes a lot longer than usual on this step. When it finally finishes the ID0 rename, it renames the rest as normal, but the folders in Nintendo 3DS are duplicated/triplicated and cross-links exist.
3. The script fails and puts out my error of "Something went wrong.\n \nSorry about that." Duplicates/Triplicates/Cross-links exist as well.

All other parts of the scripts always work without issue.

cecil_ff4_script.txt

[d0k3]

Alright, I'll need to wrap my head around that.

You say the error happens here, right? Always in the first line?

mv $[ID0path] $[ID0path]_orig mv $[ID0path]_eshop $[ID0path]

Also 2.) shouldn't be possible anymore with the new checks. Have you ever observed that issue with any other folder but the ID0 one?

[d0k3]

I just had an idea... what if you switch around the order, not processing the ID0 folder first? Does it still happen then? Is the possibly corrupted folder the first one processed?

EDIT: We'll see if this is specific to the ID0 folder this way.

[CecilFF4]

Yes, the ID0 rename lines you mentioned are where the errors occur. And no, I haven't seen the long delay problem with any other operation. It's my guess that the delay only occurs because a cross-link happened with the first mv command and tripped up the second one, though I could be wrong.

The last cross-link set of errors I got (when I reopened the issue) occurred with the various ID0 folders, the other children within the Nintendo 3DS folder, and within a freshly-created extdata for a game that was also just installed. This is the second time in recent memory that a recent, OS-created folder experienced a problem when GM9 attempted an operation on it or a parent folder. Could be a coincidence, or could be a clue. I think we should approach this from the angle of the OS not being to blame else this would be much more widespread, even amongst non-CFW users.

There was a time long ago when the ID0 rename was lower in the queue, yet I still got errors. I don't remember if they were cross-links, though, and this was many versions ago. So I'll go ahead and swap the ID0 with a later command. I don't think it'll make a difference, imho, as I've NEVER seen any errors with file operations that take place on the CTRNAND.

[d0k3]

Alright... so, it happens with a fresh folder created by the OS - that might actually be a clue. We will be getting there, but I guess it won't be fast. Also, never happens on CTRNAND. There are no other SD renames in there but the ID0 one, right?

I somewhat thought the size of the directory names could be to blame (larger names span multiple entries in FAT, thus more potential for error). Seems like that's not the case,as the names on CTRNAND are as big.

Now, one thing I stumbled over... you said the duplicates, triplicates and crosslinks do not only affect the folder you renamed an entry in (refering to the extdata you mentioned)? That should not be possible at all. If a write operation fails in a folder, only that folder should be affected. The data for any subfolders is in an entirely different data area.

You didn't actually do anything with that extdata that was affected or the folder it was contained in, right?

[CecilFF4]

I do have a script that renames a folder in my extdata for my kids' O2DS consoles, but the errors that occur on theirs are limited to the ID0 and sibling folders, iirc.

I've had errors show up regardless of the size of the ID0 currently being renamed. It could be my primary 60GB one or it could be a freshly-made almost-zero-size one.

It has happened multiple times that cross-links occur in files elsewhere on my card, though the duplicates and triplicates only occur in the Nintendo 3DS folder's children. I've had cross-links on my gm9/out NAND files, files9 files (various), and more.

You're right, it is very odd. However, if you look at the very first image I sent you, you can see errors popping up all over the card from the ID0 rename. So perhaps gm9 is writing to areas of the card it's not supposed to. And since I don't get errors on the CTRNAND (FAT16), perhaps it has something to do with the FAT32 format.

Regarding the particular extdata mentioned recently, said data was completely unusable and unrecoverable in this case. The parent and sibling folders were fine. The title data (that was also recently written to the card) was fine, too.

[d0k3]

So, does it really only happen on renames of the ID0 folder in 0:/Nintendo 3DS? If so, I may have a slight clue. You don't happen to also perform operations inside the A:/ drive?

[CecilFF4]

This problem has been going on for so long that I cannot recall if the cross-links are strictly caused by renames of the ID0. In recent memory, yes, renames of the ID0 seem to be the main cause, though other errors have happened when I have done other things, as mentioned previously (see below).

Operations I do in the A drive: copy out decrypted saves, mount apps, verify TMDs/APPs, create CIAs. These operations don't seem to be causing cross-links but may cause other errors.

However, after performing a few operations, I did eventually hit an error where a CIA creation of a 600MB file was slated to take over 30 minutes to complete (normally it's much faster than that). In addition, the title info for the selected app's TMD failed to display (it was just MK7, btw). A restart of the app allowed the title info to show, but the CIA creation still failed (by virtue of it taking too long). This is one of the types of errors I've seen in the past. In my computer, the SD revealed lost chains, another common error, though no cross-links this time. (comment from Jun 11)

[d0k3]

I meant, operations in the A:/ drive before or after the ID0 rename. Doesn't seem like there are any.

As for that strange CIA issue - well, some weird stuff happens with the 3DS SD hardware sometimes, and a lot of it has to with the SD cards themselves. The ID0 rename thing so far is the only one that's somewhat reproducable. The CIA thing is a one off occurence, I guess.

[CecilFF4]

The CIA thing happened a few times (three I think), but, like you said, not reliably reproduced. It would be mighty convenient, though, if you could reproduce the ID0-rename/cross-link issue.

[d0k3]

Okay, my slight clue turned out to be wrong. It's still weird that ID0 issue is as specific to the ID0 folder as it is. Now, to my understanding, the issue happens rarely, making it almost impossible to reproduce anything.

One thing I wonder about is if the Nintendo OS does something weird when creating that folder, something non-standard, and something that FatFS may not like. To test this, the folder would have to be created by something else (not the Nintendo OS). Moving the contents of 0:/Nintendo 3DS/ to a PC, then moving it back would be enough to prevent any influence from the Nintendo OS on that FAT dir.

This would be something intended for a long term test, ofc, as I doubt we will learn anything quick.

[CecilFF4]

Whenever I restore my damaged SD files, it's always from my PC. Doesn't stop the cross-links from happening occasionally.

Is GM9 indexing the files in a weird way? I thought that something like trying to reference a file that GM9 originally indexed in a particular way but is in fact indexed in a different way may cause cross-links. I don't know enough about read/write operations to make any better of a guess than this.

[d0k3]

Nah, all the indexing and all FAT file operations are handled by FatFS. GM9 just stores a local copy of all the dir entries, but these are only for display and reference.

[d0k3]

Okay, next suggestion. What about a stress test? I'm sure you could set up a script, maybe with multiple reboots in between. Only condition is, set up the SD card on a PC, and during the stress test, do not let anything else handle the SD card. That also means no booting of the Nintendo OS or of any other software (b9s or fb3ds as bootloaders are okay).

[Moire9]

Just giving an idea - it could be an error with the OS. That is, you have some error or corruption within a module or data file. Perhaps try doing a CTRtranafer and update?

[d0k3]

Also, next idea. Assuming the A:/ drive may be at fault, I could make a test build for you that just disables the A:/ and B:/ drives - what do you think about that? If you can compile yourself, you just need to remove / comment out these lines: https://github.com/d0k3/GodMode9/blob/master/arm9/source/filesys/fsinit.c#L37-L38 https://github.com/d0k3/GodMode9/blob/master/arm9/source/filesys/fsinit.c#L66-L67

Also, this happens on multiple consoles, @SirNapkin1334 - seems to be connected to the operations @CecilFF4 does rather than the specific console.

[CecilFF4]

Using my PC, I copied my primary SD card to a backup card sans app files (otherwise it would take a long time to set up, and the files wouldn't fit), so it was pretty similar to the original. I stress tested by doing thousands of renames of the ID0 folders. Couldn't get it to break. Also started the OS and let it re-create an ID0. Stressed. Wouldn't break.

I'm sure the point of you asking me to do this was so that I could perhaps replicate the problem more reliably; unfortunately, that is not the case. Now one big difference between this script and my others is there was no renaming of the CTRNAND folders. I know there have been no errors on my CTRNAND, but perhaps the renaming script switching between operating on the SD vs operating on the CTRNAND is part of the problem?

I can't compile your program myself, but I can use that build when I'm not planning any operations in those drives. Keep in mind it could be weeks before something could even break, though.

[d0k3]

Well, this is really hard to debug, cause it happens as rarely as it does. Operations on CTRNAND (ie. switching of the driver between NAND and SD) may have something to do with it, I still doubt it. You could make another stress test script for this.

Disabling the A:// B:/ drives won't help much, I guess. What if it doesn't happen for anotehr 4 weeks then? We won't be any wiser.

I suggest I get that release done, and afterwards I'll do a debug build for you, one that is intended to show detailed info in case something goes wrong. Maybe we can work from there.

[CecilFF4]

Ran a stress test that renamed ID0 and CTRNAND folders/files as per my original script. Ran it on a filled-up card. No break. Even ran the stress test multiple times after letting the OS create folders. No break. Damn, this issue is frustrating. Nowadays, though, I keep a very watchful eye on exactly what I do before I engage in "risky" behavior (namely renaming stuff).

[d0k3]

@CecilFF4 - just a quick heads up - this issue is not forgotten. I'm still looking into doing that debug build, with a mistery like this one it's just not easy to tell what exact info could help. Did it happen again since the last time you mentioned it here?

[CecilFF4]

No it hasn't happened again, though I've been trying to not use my renaming scripts precisely because I don't like having to take the time to fix my SD.

[CecilFF4]

It happened again. I was using gm9 1.7.1 (public release). I was playing an NDS game. Closed the game but did not let it boot to home. Just booted to gm9 directly. Did my standard rename script and got cross-links and other stuff. New pic attached. Two things to add: I have no idea how the volume creation date was set to 60 years in the future (this could've happened separate of this issue), and my rename script did have the swapped ordering that you suggested a while back. That is to say that my ctrnand renames occurred first, ID0 last. The ID0 is still where the error occurred.

[d0k3]

Okay, I guess we can at least say that issue is limited to the 0:/Nintendo 3DS/ folder. That's weird. If it's only there, FATFS can't be at fault. It's the SD card driver (highly unlikely) or it's something in the GM9 code.

Can you show me the script, as it is now?

[CecilFF4]

Here you go. Yes, it's limited to the Nintendo 3DS folder, but that's the only location that I do renames on my SD card. If I renamed other items on my SD, the issue might present itself. Not sure. I'm gonna toss a rename in there for something else and we'll see.

rename script.txt

[d0k3]

We can at least safely say it does not happen for NAND. And, FatFS does not care about the source of the data, it's all the same for it.

It would actually help a lot if I knew if that can happen outside of the 0:/Nintendo 3DS/ folder, so the additional rename will be very helpful.

Also, from what I understand, you did nothing else in GM9 this time, right? Just ran the script immediately.

EDIT: You're not running from a scriptrunner, right? Just plain old GM9?

EDIT2: It may be a good idea to check if $[ID0path]_orig (and all the other orig paths) exist before starting. Maybe offer deletion in that case.

EDIT3: (last one, promised) It's maybe a good idea to put that additional rename before the ID0 rename. Rename something uncritical, of course. Do you happen to recognize some pattern in which folders are affected? Seems like the PRIVATE folder always is. Also, what's that EBNTR folder?

[CecilFF4]

Yeah, did nothing else. Immediately booted gm9 after using an NDS game, immediately ran my script, almost immediately saw my error message that something went wrong.

I don't know what a scriptrunner is. Sounds like something that runs scripts, but I didn't think anything other than gm9 could run gm9 scripts... ?

All the paths exist before I start. I have a check early in the script for making sure everything is named properly, including the ID0 folders.

I've already implemented the third edit. I'm renaming the JKSV folder I had from the previous version (named JKSV_old). It's got a lot of files in it, much like the ID0, so probably the most similar to the ID0 I have on my SD. Plus it's a folder I already got archived on the computer, so I don't care if it gets messed with.

I think I mentioned it before, but all the child folders inside the Nintendo 3DS folder get messed up on some ID0 renames. That means all my ID0s (I have three different profiles), my private folder, and that EBNTR folder, which I think is just a relic from an old BootNTR. Probably deletable, but I haven't gotten around to it.