The RTC maybe set to UTC instead of local time

[skullandbones]

I note that the FAT command is using the RTC time and date to write files to the FAT32 file system on the SD card or CF card aka media.

This is problematic because modern computers may display the file's timestamp based on the timezone. This is back to the problem of computers using local time or UTC as their clock source, such as Windows versus UNIX (Linux).

This is easy to demonstrate by setting the RTC to BST (UTC+1) which is local time and use the FAT command to write a file to the media. FAT DIR 2: shows the file with an expected time that matches the RTC. However, now put the media into a modern computer, does the OS assume local time or UTC as the timezone of the file? Linux assumes the file's timezone is UTC so will adjust the timestamp for timezone handling for display purposes and add 1 hour to the timestamp so showing the wrong time.

I expect CP/M does not have any support for timezones which is fine. Trouble comes when inter-operating with modern OSes.

The other problem is day light saving (change of timezone). If the RTC is set to UTC then day light saving can be ignored. If the RTC is local time then the RTC will need to be adjusted twice a year.

May I suggest that a note is added to the user guide to indicate this potential issue. Sometimes, it may be preferable to set the RTC to UTC rather than local time because that can improve interoperability with systems such as Linux systems.

Thanks, Dean

[wwarthen]

Very interesting. I mostly use Windows, so I have not seen this problem. Definitely worth a note in the documentation. I will do that soon.

Thanks,

Wayne

[skullandbones]

@wwarthen let me do some more investigations before you update the user guide.

The dependency might be which version of FAT32 is being used. I think there is more than one file system type ID that can be used in the partition table for FAT32.

In your partition table image you are using FAT16 which means other external tools on modern computers would be rewriting the partition table to make it FAT32.

I do transfer files between Windows and Linux on FAT32 media and I don't see the shown file timestamps jump around between the two OSes.

I am in the UK so half of the year, my local time is UTC so this issue is not really an issue for me.

Thanks, Dean

[wwarthen]

Thanks @skullandbones. It does sound like there is more to this than I ever realized. I will be interested in what you find out.

I will do a little research on my end regarding the modern OSes switching the FAT filesystem from FAT16 to FAT32. I'm not sure why that is happening.

Thanks,

Wayne

[wrljet]

That's a pretty intrusive thing to do. And would be complex and dangerous to do in-place. I've never heard of a pgm doing that.

[wwarthen]

Bill, yes, doesn't seem reasonable.

What I am seeing in my initial testing is that when Windows is asked to format the partition, it defaults to 0x0E. This is the LBA-only variation of 0x06. I don't see Windows changing the partition type id of a partition that is already formatted. The RomWBW images use partition type id 0x06.

So, it might be better for the RomWBW images to use 0x0E so that reapplying the combo image does not change the partition type id back to 0x06 if the partition was originally formatted using Windows.

I am not sure what Linux is doing.

By the way. Just found a problem in the RomWBW FAT application. The FAT FORMAT command is misbehaving due to a change in the HBIOS Disk Device API call. It is overwriting the CP/M partition -- ugh! This will be fixed shortly.

Thanks,

Wayne

[skullandbones]

@wwarthen I hope that I did not confuse you.

Your hd1k_prefix.dat has the following partition table.

pi@raspberrypi:~/projects/romwbw/Binary$ fdisk -l hd1k_prefix.dat
Disk hd1k_prefix.dat: 1 MiB, 1048576 bytes, 2048 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x9921862d

Device           Boot   Start     End Sectors  Size Id Type
hd1k_prefix.dat1         2048 1050623 1048576  512M 2e unknown
hd1k_prefix.dat2      1050624 1837055  786432  384M  6 FAT16

This is OK for 1GB media. Problem comes when the partition exceeds 4GB (4GiB) in size because FAT16 is limited to a maximum volume size of 4GB (4GiB). Therefore, FAT32 is needed for greater than 4GB (4GiB) volumes,

Linux can use FAT16 and FAT32 by using the vfat driver.

I used a Linux program called gparted which is a GUI program that allows partition tables and file systems in partitions to be manipulated such as resizing. I used gparted to replace the FAT16 partition with a FAT32 partition that used the rest of the media space > 4GB (4GiB). Therefore, gparted selected the partition ID type for FAT32 and gparted formatted the FAT32 file system.

I hope that clears thing up ?

Dean

[skullandbones]

@wwarthen as per our previous discussions, once the partition table and slice metadata area has been written. The user should write slices to partition 1 and not to the start of the media. That will avoid the partition table from being over-written.

Alternatively, the user could read off the first 1MB (1MiB) of the used CP/M media to create their own version of hd1k_prefix.dat and use that data to create their own combo images. That would avoid corrupting or modifying the partition table.

[skullandbones]

@wwarthen getting back to the topic of system clocks.

When using a PC to dual boot into Windows and Linux (only 1 OS runs at a time), there is a historical fight over whether the system clock is set to local time or UTC. Windows wants local time and Linux wants UTC.

So this same fight is now playing out with CP/M versus Linux for interoperability.

Here are some of my test results:

I created 3 files on an SD card on a FAT32 partition.

FRMCPM.TXT was created on a CP/M machine at 18:09:32 BST (UTC+1).

FRMLINUX.TXT was created on a Linux machine at 17:56 BST (UTC+1)

FRMWIN.TXT was created on Windows 7 by doing 'dir f: > f:\FRMWIN.TXT hence 0 size
was recorded inside the file's contents and was created at 21:06 BST (UTC+1)

Note that I had to delete the CP/M partition #1 partition entry from the partition table to allow
Windows to use the FAT32 partition. That was a crazy limitation that Microsoft added to not
allowing multiple partitions on removable media.

Using CP/M to show file timestamps
==================================

SD card directory listing using CP/M

C>FAT DIR 4:

Directory of 4:

10/12/2023  18:11:30           128  ---A  FRMCPM.TXT
10/12/2023  16:56:44           551  ---A  FRMLINUX.TXT
10/12/2023  21:06:28           469  ---A  FRMWIN.TXT

CP/M correctly reports the time of FRMCPM.TXT as 18:11:30 local time BST (UTC+1)

CP/M incorrectly reports the time of FRMLINUX.TXT as 16:56:44 (actually UTC)
which should of been 17:56 so 1 hour early.

CP/M correctly reports the time of FRMWIN.TXT as 21:06:28 local time BST (UTC+1)

Using Linux to show the file timestamps
=======================================

ls -al
-rw-r--r--  1 pi   pi    128 Oct 12 19:11 FRMCPM.TXT
-rw-r--r--  1 pi   pi    551 Oct 12 17:56 FRMLINUX.TXT
-rw-r--r--  1 pi   pi    469 Oct 12  2023 FRMWIN.TXT

Linux incorrectly shows the time of FRMCPM.TXT as 19:11 (actually UTC+2)
which should of been 18:09 so is 1 hour late.

Linux correctly shows the time of FRMLINUX.TXT as 17:56 BST (UTC+1)

Linux incorrecly shows the time of FRMWIN.TXT as 2023 (time is in the future so shows the year instead)
which should of been 21.06

Now using Windows 7 to show the file timestamps
===============================================

FRMWIN.TXT contains

 Volume in drive F has no label.
 Volume Serial Number is FD8E-70A3

 Directory of F:\

12/10/2023  18:11               128 FRMCPM.TXT
12/10/2023  16:56               551 FRMLINUX.TXT
12/10/2023  21:06                 0 FRMWIN.TXT

Windows correctly shows the time of FRMCPM.TXT as 18:11 local time BST (UTC+1)

Windows incorrectly shows the time of FRMLINUX.TXT as 16:56 (actually UTC)
which should of been 17:56 so 1 hour early

Windows correctly shows the time of FRMWIN.TXT as 21:06 local time BST (UTC+1)

As can be seen, Windows and CP/M will interoperate OK as long as both machines are using the same local time. However, this is a bit of a mirage because if one of those machines is moved to a different timezone (including daylight saving) then trouble comes because the timezone is not recorded in FAT16 or FAT32.

This can be seen when using a camera with a FAT32 SD card and taking photos in a different timezone. When you come home there is likely to be an expectation that the file's timestamps show the time when the photo was created. But Windows can only show the timestamps as local time.

Linux and UNIX systems in general use UTC for all time references. This allows those systems to interoperate across timezones by every machine using the same UTC clock reference as the baseline. Therefore, Linux manipulates what the user sees in looking at file listings to adjust the timestamps for timezone and daylight savings. The actual timestamps on the media are not modified.

I will endeavour to get some results by setting the CP/M clock to UTC.

Thanks, Dean

[wwarthen]

This is OK for 1GB media. Problem comes when the partition exceeds 4GB (4GiB) in size because FAT16 is limited to a maximum volume size of 4GB (4GiB). Therefore, FAT32 is needed for greater than 4GB (4GiB) volumes,

Linux can use FAT16 and FAT32 by using the vfat driver.

I used a Linux program called gparted which is a GUI program that allows partition tables and file systems in partitions to be manipulated such as resizing. I used gparted to replace the FAT16 partition with a FAT32 partition that used the rest of the media space > 4GB (4GiB). Therefore, gparted selected the partition ID type for FAT32 and gparted formatted the FAT32 file system.

I hope that clears thing up ?

Yes, now I understand the scenario. This makes sense.

Thanks,

Wayne

[wwarthen]

@wwarthen as per our previous discussions, once the partition table and slice metadata area has been written. The user should write slices to partition 1 and not to the start of the media. That will avoid the partition table from being over-written.

I understand the problem you are worried about. However, most users are not going to understand the tools and techniques for doing this. My thinking is that the "standard" prefix (with partition table) would work very well for virtually all users, so there would be no need to change the partitioning. I sort of expect that anyone that modifies the partition table would understand what they are doing and be able to handle the implications.

My intent was to make it clear in the User Guide that if you change the partition table, you can no longer just copy over the standard prefix and/or combo image. I may not have made this as clear as I should (I will review this). I also acknowledge that I need to finish updating the Hard Disk Anatomy document to include the hd1k layout.

Based on the last couple years of user feedback, the standard prefix works great for almost everyone. In fact, it was specifically created based on requests from many users who don't want to deal with setting up the partition table. Having 64 slices and a 384MB FAT filesystem is plenty of space, yet fits nicely on any CF/SD Card claiming to support 1GB. Other than you, I only know of one other user that had any interest in modifying this standard setup -- and he was very competent to handle all of the resultant complexities.

Alternatively, the user could read off the first 1MB (1MiB) of the used CP/M media to create their own version of hd1k_prefix.dat and use that data to create their own combo images. That would avoid corrupting or modifying the partition table.

This is exactly the kind of thing I would expect an advanced user (like you) to do. As soon as you start to go beyond using the standard prefix, there are many ways to manage your storage and backups. I felt like it would be very hard to document the different ways to handle a non-standard prefix/partition table.

One last comment about expanding the FAT partition. The FAT application is rudimentary compared to actual MS-DOS, Windows, Linux, etc. As a result, it starts to get very slow as the size of the FAT filesystem increases. I don't recommend going beyond about 1GB unless you have a very specific use case for it

Thanks,

Wayne

[wwarthen]

@wwarthen getting back to the topic of system clocks.

When using a PC to dual boot into Windows and Linux (only 1 OS runs at a time), there is a historical fight over whether the system clock is set to local time or UTC. Windows wants local time and Linux wants UTC.

So this same fight is now playing out with CP/M versus Linux for interoperability.

Linux and UNIX systems in general use UTC for all time references. This allows those systems to interoperate across timezones by every machine using the same UTC clock reference as the baseline. Therefore, Linux manipulates what the user sees in looking at file listings to adjust the timestamps for timezone and daylight savings. The actual timestamps on the media are not modified.

Very thorough testing. Your results are exactly as I would expect. Not sure there is much to be done about this except a note in the User Guide about the fact that the CP/M has no concept of timezone when dealing with date/time.

Thanks,

Wayne

[wwarthen]

Very thorough testing. Your results are exactly as I would expect. Not sure there is much to be done about this except a note in the User Guide about the fact that the CP/M has no concept of timezone when dealing with date/time.

Actually, it is more accurate to say that neither CP/M nor FAT filesystems have any concept of timezone.

Thanks,

Wayne

[skullandbones]

@wwarthen here are some results using CP/M with the RTC set to UTC.

BSTLINUX.TXT was created on Linux at 17:00 BST (UTC+1)

UTCCPM.TXT was created on CP/M at 15:57 UTC

Using CP/M to show the file timestamps
=======================================

C>WDATE
Saturday 14 October 16:05:15 2023

C>FAT DIR 4:

Directory of 4:

10/14/2023  15:57:18           128  ---A  UTCCPM.TXT
10/14/2023  16:00:36            52  ---A  BSTLINUX.TXT

It can be seen that there is an expected gap of 3 minutes between the files.

The timestamps are displayed in UTC as expected.

Using Linux to show the file timestamps
=======================================

ls -al BSTLINUX.TXT UTCCPM.TXT
-rw-r--r-- 1 pi pi  52 Oct 14 17:00 BSTLINUX.TXT
-rw-r--r-- 1 pi pi 128 Oct 14 16:57 UTCCPM.TXT

It can be seen that there is an expected gap of 3 minutes between the files.

The timestamps are displayed in BST (UTC+1) as expected.

That test appears to confirm that setting the CP/M system clock to UTC allows the FAT command to interoperate with Linux to show consistent timestamps. Just need to remember that the 2 machines are in different timezones ;-)

I expect this issue also impacts macOS as it is UNIX based. But I don't have a MAC so I can't check that.

Anyway, I think it would be helpful to add a comment into the User Guide with some advice on using local time versus UTC.

[skullandbones]

@wwarthen

This is exactly the kind of thing I would expect an advanced user (like you) to do.

My day job is a Linux kernel software engineer working on embedded Linux systems. I use a company Linux desktop environment based on Ubuntu so I don't use Windows. In fact, Microsoft is more and more supporting Linux in the business arena. I can use Office365, MS Teams and Outlook on Ubuntu Linux. Also, Windows itself has WSL (Windows Sub-system for Linux) that allows a Linux desktop to run in parallel with Windows.

I never had the opportunity of playing with CP/M in the 1980s. I think our school had a RML Research Machines 380Z https://en.wikipedia.org/wiki/Research_Machines_380Z. I learnt Z80 Assembly Language on my Sinclair ZX Spectrum 48K https://en.wikipedia.org/wiki/ZX_Spectrum.

I am interested in trying to use the native CP/M build environments to see what they can do.

Thanks,

Dean

[wwarthen]

That test appears to confirm that setting the CP/M system clock to UTC allows the FAT command to interoperate with Linux to show consistent timestamps. Just need to remember that the 2 machines are in different timezones ;-)

I expect this issue also impacts macOS as it is UNIX based. But I don't have a MAC so I can't check that.

Anyway, I think it would be helpful to add a comment into the User Guide with some advice on using local time versus UTC.

Thanks again for all the testing Dean. Super thorough.

Yes, I am going to leave this issue open until I add some comments in the User Guide. I don't plan to go into the level of detail that you have, but I will be linking to this discussion for those that want them.

Thanks!

Wayne

[wwarthen]

My day job is a Linux kernel software engineer working on embedded Linux systems. I use a company Linux desktop environment based on Ubuntu so I don't use Windows. In fact, Microsoft is more and more supporting Linux in the business arena. I can use Office365, MS Teams and Outlook on Ubuntu Linux. Also, Windows itself has WSL (Windows Sub-system for Linux) that allows a Linux desktop to run in parallel with Windows.

Although I am a Windows programmer by trade, I am a big fan of the Linux movement and personally hope it is the future. I am committed to ensuring that RomWBW remains as compatible with Linux as possible (especially software builds). There will be, unfortunately, some areas (like date/time) where CP/M lacks the sophistication to interoperate with Linux seamlessly.

I never had the opportunity of playing with CP/M in the 1980s. I think our school had a RML Research Machines 380Z https://en.wikipedia.org/wiki/Research_Machines_380Z. I learnt Z80 Assembly Language on my Sinclair ZX Spectrum 48K https://en.wikipedia.org/wiki/ZX_Spectrum.

I am interested in trying to use the native CP/M build environments to see what they can do.

It is hard to explain why, but I get far more pleasure from programming on these older platforms than I do on modern ones. I think it has to do with how close you are to the actual machine.

Thanks,

Wayne

[wwarthen]

Closing this issue as the User Guide has been updated to include a section on the Real Time Clock.

Thanks,

Wayne