File-browsing: Missing ".." in certain situations

[sy2002]

Deft reported via Skype, referring to this YouTube video: https://www.youtube.com/watch?v=a7FVWHwZxvo

Steven is having trouble, he is not seeing the parent folder ".." dots.

I consider this a bug at this moment and I am assigning it to myself for further investigation.

[sy2002]

@paich64 Do you have an update for me?

[paich64]

@sy2002 this is the last feedback i got from @retrocombs : "It is and appears as if the Mac ._ files might be the issue. I've not removed them to check it out, but that's my next step. I'm also happy to send you a disk image, but might be premature. Let me check my solution first. Might be a few days." I have asked him to provide an update and also asked again if he could share his SD card image.

[paich64]

@sy2002 @retrocombs just told me he had forgotten about it and that he'll try to get in touch with me about it within 1 or 2 days.

[sy2002]

@paich64 Here is my status:

I created a folder called TEST DIRECTORY that has these "dot underscore files". I placed it in the root folder of the SD card and in to the /c64 folder. In both cases everything worked perfectly fine as shown for example on this screenshot (the .. "folder" to move upwards is there):

What I did:

I manually created a folder with these "dot underscore files" because the ZIP file provided by RetroCombs (this here: TEST_DIRECTORY.zip) did not contain them.

Bottom line: I am not yet able to reproduce the problem.

Next step: I will try the SD card image provided by Retro Combs.

[sy2002]

Update: Using the SD card image provided by Retro Combs, I can reproduce the issue.

[sy2002]

@paich64 This is not a bug and it is also not related to macOS' "dot underscore" files, so I will close this issue. Instead, my thorough investigation (see below) has shown, that the SD card image provided by Retro Combs contains a corrupted structure in the folder 2023-11-12 Livestream. Please communicate with Retro Combs and/or show him the detailed report below.

I am aware, that MacOS and other operating system can live with the corrupted structure that I discovered, i.e. this folder does not generate any problems there. But still, the structure is corrupted and implementing a workaround for this would cost an unreasonable amount of time and effort on my side while there is a simple workaround to fix the corruption which is why I decided to not do this right now.

If an issue like this starts to happen more often and/or more users are affected, I will consider to implement "robustness" features to the QNICE FAT32 implementation that will then be able to "live with" these kinds of corruptions (and others) and "just work".

@MJoergen FYI.

Detailed report

The issue that was reported is, that a certain folder called 2023-11-12 Livestream is not showing any <..> "dotdot" element so that as soon as you navigate into this folder, you cannot "leave" this folder any more.

First of all: Yes, using Retro Comb's SD card image on my local SD card, I can confirm that this symptom occurs also on my machine, when I navigate to this folder.

But what is interesting is, that the disk image provided by Retro Combs was "cleaned up", because the "dot underscore" files that MacOS creates and that are shown in the screen shot above, are gone in the meantime. So I assume Retro Combs did an experiment and deleted the "dot underscore" files to check, if the issue goes away.

My finding is: It does not. The issue persists in this very folder.

I mounted the SD card image on my Mac and here is what the command ls -la outputs:

One can see, that the folder 2023-11-12 Livestream - which is the very folder that suffers from the issue - is not having these "dot underscore" files (any more). But another folder, called C64 Games does indeed have the "dot underscore" files.

Sure enough, the C64 Games folder can be browsed without any issues using the C64 core (i.e. it DOES show the <..>), even though it does contain "dot underscore" files.

This proves the theory wrong, that the "dot underscore" files are related in any way with the problem that we can observe in the 2023-11-12 Livestream folder, which indeed does not show any <..>.

In a next step, I modified the code in the firmware that iterates through all FAT32 entries in a folder in dirbrowse.asm in a way that it does NOT ignore hidden files any more. And boom: The <..> appeared in the folder 2023-11-12 Livestream. This is the very corruption: The .. FAT32 entry in the folder 2023-11-12 Livestream is having the "hidden" flag. This is not allowed and therefore a corruption.

I have no idea how it could happen on Retro Combs' machine and/or SD card.

Workaround: Use macOS Finder and just duplicate the folder using the context menu (see red arrow here):

After the Finder duplicates the folder, delete the old one. The new one will work, i.e. Finder will fix the corruption while making a copy.

[sy2002]

Reopened, because on 1/28/24 I learned that this issue happened again. This time another user was affected. See screenshot from Discord. TL;DR: I plan to tackle this for V6

[cybern0id]

Dr_Vulcan in the discord is me!

I've used a Windows VM to inspect the SD Card and according to both File Explorer and the attrib command in the shell, all files and folders have the A = Archive flag set. However, this is not shown using ls -alhO on macos where it only shows the ".." "file" set as "arch". In other words, it is almost as though macos reports the opposite of Windows for file attributes. Using the Windows VM, none of the folders or files have the H = Hidden bit set. Unfortunately, it is not possible to change the file attribute bits for the ".." file/folder using Windows as far as I can tell.

As a test, I tried creating new folders in the root of the SD card using both macOS (using terminal and Finder) and Windows File Explorer. Attempting to open these empty folders using the C64 core Help Menu \<Mount Drive> option shows some interesting results: Opening the folder that was created in Windows results in showing an empty folder with the ".." allowing return to parent directory. Opening the folders that were created in macOS, using both Terminal and Finder, results in the "Nothing to browse - Press Space to continue" information window appearing. It seems like macOS doesn't create new folders in a compliant manner on FAT32 file systems.

[cybern0id]

I've just done some checks using a hex editor and disk explorer to compare the FAT directory table entries for each of the newly created directories on the SD card. It does appear that macOS sets the directory attribute bit to 'hidden' in the directory table for both "." and ".." inside the new directory when creating it, unlike Windows: At byte offset 0x0B in the directory table, the value is set to 0x12 rather than 0x10. Using a hex editor to change the value to 0x10 results in the macOS created directory working as expected. I used the Wikipedia page for the FAT filesystem to help me understand. I still haven't found a way to unset this flag using CLI tools yet though.

[cybern0id]

I'm looking at

https://github.com/MJoergen/C64MEGA65/blob/442c338902786eeba32446406e3e377a03c0591a/M2M/rom/dirbrowse.asm#L83

and wondering about

https://github.com/sy2002/QNICE-FPGA/blob/5ecc7b7f8f9d93cb267ef6700682c8e010603148/monitor/sysdef.asm#L355

[sy2002]

@cybern0id / Dr_Vulcan: Thank you for your analysis and for the workaround! Really interesting to hear that the problem occurs on your machine kind of EVERY time you create a new folder using macOS? Can you reproduce this with a newly formatted FAT32 SD card, too? What macOS version do you run? I am running "Big Sur" and there the problem never occurs. If you run a newer version then it might be that Apple added this "feature" to macOS and this might mean that I might need to re-think my prioritization of this problem because if it happens for all users of newer macOS versions then waiting 12 months for a fix might be too long. (I am talking about fixing this in version 6 of the C64 core, planned for next year vs. version 5.1 which is scheduled for "soon".)

The issue is, that I cannot just change the file attributes that you pointed out in line 83 of dirbrowse.asm: It would then show all the nasty hidden files in each folder such as macOS "dot files" and other "crap". So indeed I do want to continue to use FAT32$FA_DEFAULT in general. But I might need to do one additional iteration. Something like: "If I did not find any .. during my first pass of the current directory, then search for the .. without filtering for attributes and then add this to my linked list of files" Or - and I need to check if I "know" this while being in this routine: If I am not in the root folder, then always add ".." to the linked-list in case it is not already there. This would be even simpler.

Since it seems you can reproduce the problem easily: Would you be willing to test a fix, as soon as I come up with something?

[cybern0id]

I don't mean to put pressure on you to fix this issue sooner rather than later; you (and MJoergen) have already done a seriously amazing job building this core and it is easy enough to work around. I thought my findings might help with a solution one day :)

In any case I am more than happy to help in any way I can, including testing. To confirm the issue happens repeatedly, I have just quickly done the following.

Opened a brand new Samsung 32GB Pro Endurance UHS-1 class 10 microSDHC card bought via official channel on Amazon which is pre-formatted FAT32. For my first test, I have left this formatting as is. Created 3 new directories; the first via terminal, the second via Finder and the third via terminal, whilst also creating a subdirectory in this third one: MACTEST1 MACTEST2 MACTEST3 > MACSUB1 Using macOS 'Hex Fiend' app, I opened the SD card device and searched for MACSUB1. The FAT directory table shows that indeed, both . and .. are created with attribute byte set to 0x12 rather than 0x10. See this screen cap for reference:

I'm using a 2022 13" Macbook Air M2 with macOS Sonoma 14.3.

Apple do have a habit of doing strange things and changing stuff without telling anyone.

[sy2002]

@cybern0id THANK YOU for this very thorough and unambiguous analysis. So my theory is: Between the macOS Big Sur version that I am using, which is Version 11.6.8 and the macOS Sonoma version you are using which is 14.3, Apple changed the way they treat . and .. when creating folders.

Being a true Apple fanboy (sigh) I cannot let down all the Apple users who use new OS versions and therefore this is enough reason for me to fix the issue in V5.1 :-)

[cybern0id]

I've just confirmed that this still happens when formatting the SD Card using Disk Utility, both MSDOS FAT32 MBR and exFAT MBR, and then creating the same folder structure. The results are also the same after formatting the card using Linux, fdisk to delete and create new DOS MBR and a new 0x0b W95 FAT32 partition, then formatting with mkfs.msdos, then creating the new directories back on the Mac.

[sy2002]

@cybern0id Here is version Alpha 13 of the core which should fix the issue. Can you please confirm and document the result here in this issue? C64MEGA65-V5.1A13.zip

[cybern0id]

Thank you @sy2002! You have indeed fixed the problem. I tried loading .d64 and .prg files from multiple different and varying levels of nested directories and it all works. I (and I'm sure many others) will really appreciate you taking the time to sort this.

[sy2002]

Thank you @cybern0id for your support! Now that it works, we can close the issue.