Details on nature of Feather M0 failure, and boot-strapping circuit

[flimshaw]

Hey there, amazing work, I really want to get it running.

I was curious, for the M0 no longer working, is there any symptom I can check? Should it fail when flashing to the board, or does it just silently fail during the normal process?

And secondly, in your notes, when you say "a momentary button to 3.3v and gpio pin 6", when exactly do you hit that button and when? It seems like with a 10K resistor to ground on pin6, my M0 is trying to send data but I'm not picking it up on the M100.

Thanks!

[flimshaw]

Update: with a 10K to ground on pin 6 alone, I am able to bootstrap/install TS-DOS and get mostly to the end. after the '.....' fill up, it flashes 'DONE', it crashes with an ?SN Error in 3. But, if I run the BASIC oneliner in TMP.DO, TS-DOS launches!

When I attempt to go to DISK, the top header loads and it just sits there forever. If I reset the M0, the M100 throws a 'Communications Error' and fails to TS-DOS gracefully.

Anyhow, progress but still no luck. I've run through it a couple times so far with the same results.

[flimshaw]

Update the last: I re-built with "ENABLE_SLEEP" commented out, and now it's working on my Model 100 w/ Feather M0 + generic 4-pin RS-232 adapter with the jumper pins as you describe.

[bkw777]

For bootstrap, (from any progam in general not just pdduino) any time you get where it looks like it's sending the file ok but then the program crashes on the receiving side, increase the per-character sleep in the bootstrap sender. On PDDuino, the setting is LOADER_MS_PER_BYTE in cfg_behavior.h

It's 6 right now, increase it to 8.

It's 6 right now, but that is an old value and while working on dlplus and trying a lot of other BASIIC files on all the different machines, I discovered the necessary value is variable depending on the BASIC code being sent. Some programs need more than others. For a long time an old version of ts-dos and teeny loaders worked at 5ms, old versions of dlplus and mComm etc all used 5ms.

But I started using the function for other things like REX and REXCPM setup and loading various things from the M100SIG archive, and I refactored some of the existing tpdd client loaders to make them smaller, and along the way kept finding things that needed a bit more, and the current "high water mark" is 7ms, and so dlplus and pdd.sh now use 8ms.

I just never updated pdduino yet to match because really pdduino needs a major overhault to work like dlplus. (my dlplus not the original) pdduino is currently getting a lot of things wrong, and so a lot of clients choke on it. By "clients" I mean any code that interacts with the drive from the client side, not just ts-dos or teeny. For instance the on-the fly loader code in ultimate rom II, the dictionary code in Sardine, etc.

A few months ago I did a lot of work on dlplus, and pdd.sh too ffrom the client side. dlplus is now pretty solid and serves as a good reference. pdduino was really just a "barely works in perfect conditions in one case" that still needs more work to get actually good. And now that I have dlplus as a good reference, the way to fix up pdduino so it actually does all the right things is to model it after dlplus.

Be aware that with sleep totally disabled, it's burning 20-40ma at all times. sleep drops it to 2-3ma 99% of the time and just spikes to 20-40 during activity.

Sleep relies on a hardware interrupt on the RX pin. Is it working reasonably other than having to disable sleep? If so then that's great because that's something I can actually investigate instead of just "IDK used to work".

[bkw777]

I just updated it.

[flimshaw]

Thanks for all that info, super interesting. I was going to try out dlplus but couldn't get my ~12 year old usb to serial adapter working. I'm waiting on a Rex#, but just stock machines w/ full ram upgrades at the moment.

So it is indeed working pretty well. I've got a 100 and a 102, the 102 has always been more finicky with serial but both were able to bootstrap TS-DOS with the resistor trick. The installer still crashes at line 3, and it looks like, for whatever reason, the ? characters in TS-DOS.100 are not coming through properly? They are in the source file, but it seems like BASIC maybe just gives up on them after a while? Or there's a typo I'm not seeing.

But, i confirmed that in both machines, and had maybe 8 otherwise-successful bootstraps with the same behavior, so it feels like not a random rs232 glitch/overflow. Otherwise, as long as the batteries are charged, I can save/load .DO files with ease. I was loading .BA files that were, in fact, source files at first, which was causing the constant crashing 🤦 .

[flimshaw]

As for porting over your work on dlplus, are there any exotic/large libraries it's using? Or does it need much memory? Sometimes you can get lucky with C and have it easily port over to Arduino.

So i have no idea what I'm talking about BUT: I wonder if all your stand-alone functions like checksums and stuff would come right over? You would def need patching/rewrite for rs232 and storage, and depending on how that's wired up you could hit memory issues. The M0 only has 32KB of ram, so might need to get clever with file streaming etc.

Anyhow, I could/probably am way off, but I'd be down to assist/test where possible. Think dlplus could get forked into something that could build for arduino/teensy?

[bkw777]

No libraries, just plain c, but it's structured just differently enough from pdduino that it looks like it would be easier to port it fresh than to start with existing pdduino, except maybe the board-config framework.

The logic should port fine, but merely with some fluff stripped, almost none of the debugging and server-side display, obviously anything to do with the serial port is different, and probably not the full dlplus feature set because dlplus now supports sector access on disk images, and two forms of that because tpdd1 & tpdd2 have totally different sector access.

That's got to be pushing the available rom/ram considering we're already getting up there.

It doesn't use a lot of ram in the sense of, it still works packet-at-a-time. Each packet, reading or writing, is processed fully to or from disk before acking and allowing the next packet, and the max payload of any single packet is only 128 bytes. So no problem there.

But for instance the way pdduino handles the checksums kind of permeates everything because the "write-to-serial-port" function adds to the checksum for every byte passing through the function, and then because that is not actually right there has to be special exceptions here & there to send bytes that aren't part of a checksummed range. And now I don't want to do it that way.

Then again, I may be in the wrong when it comes to targeting this environment. That checksum-as-you-go method was made by the original SD2TPDD author Jimmy Petit, and another thing he had originally which is no longer in here is he used a ring buffer for most everything, and it was kind of complicated keeping track of how it worked so me and Jim Brain replaced that with ordinary variables and buffers that get cleared and start from zero each time. Maybe the original way was actually the more proper embedded target way to do things because maybe it uses less ram and less cpu, for instance you don't really have to zero the buffer necessarily, just always move forward and just remember your current start and stop addresses, and zero any individual bytes that need it as you go, and basically just "be careful". So maybe his approach to the checksum was better too for this kind of environment. Maybe not universally I mean but maybe it was a case of doing the math that if most bytes do need the checksum (they do, by far), then treating the non-checksum bytes as the exception may use less cpu all around, even if it's a less convenient mental model.

Never the less, I will probably try to do it just like dlplus rather than like original SD2TPDD if I can get away with it, even if it might not be the leanest possible way, it will be the easier-to-get-right way. Which should be fine because it's like half way there now already. And maybe it's not really any leaner anyway.

[bkw777]

Increasing that per byte sleep to 8 didn't fix it? Bump it to 10, or 20, just to prove you're actually getting the new code onto the M0. Even 10 will be unnecessarily slow, but just as a diagnostic step to prove it's taking effect try even a ridiculous value just to prove it works even if stupid slow, and then back down until it fails again.

Also, would you mind trying dlplus with the same loader file? Or, if you have windows instead of linux or mac, could you try https://github.com/bkw777/tsend ? Or, there is also a bootstrap function in pdd.sh which only needs bash, but on mac in currently needs bash from macports or brew or maybe nix I haven't tried that, and those require installing xcode so, it's really no easier to use pdd.sh than to use dlplus on mac until I massage it down to get it to work in the stock bash.

Basically all of those also work well, and work the exact same way, but, different hardware, different environment etc.

It's important also to be using the right serial wiring including the dtr/dsr lines. Follow http://tandy.wiki/Model_T_Serial_Cable

I'm curious if it's maybe something with the 100 or the serial adapter. That same loader file works fine for me, but with dlplus and my own different usb adapters and serial cables. I haven't tried with pdduino lately. But I can't imagine what pdduino could do differently. It's just sending bytes and sleeping the same as all the other versions of that same routine in tsend.ps1 and dlplus etc, and they all work.

The loader file is definitely more "difficult" for the 100 to swallow now than it used to be, since it used to work with old tpdd servers with only 5ms per byte. But, that doesn't really matter because the bootstrapper still needs to target the worst case not the best case. It needs to successfully send any valid BASIC code period, even if 99% of the time it's only used for a loader that happens to be "easy" and could have worked with a much smaller per-byte sleep. The user needs to be able to trust that the data is 100% correct, not merely it seems to be ok because it didn't seem to crash.

Another thing you could try just see what happens is get an old version of the ts-dos loader from the previous release of dlplus. https://github.com/bkw777/dlplus/tree/v1.5.010 https://raw.githubusercontent.com/bkw777/dlplus/v1.5.010/clients/ts-dos/TS-DOS.100 Same for any other loaders you want to try because I modified a bunch of them all around the same time.

If it works, that wouldn't really be fixing the problmem just avoiding it. Basically the bootstrapper process would still be undependable, merely you'd be "lucky" using a file that doesn't expose the problem. You could never trust the data. If the corruption didn't happen to cause the program to crash, you'd never know some of the data was wrong. I would definitely not be satisfied with this even if it works.

[go4retro]

The logic should port fine, but merely with some fluff stripped, almost none of the debugging and server-side display, obviously anything to do with the serial port is different, and probably not the full dlplus feature set because dlplus now supports sector access on disk images, and two forms of that because tpdd1 & tpdd2 have totally different sector access.

With a bit of work, I think one could write a piece of code that would work on both targets, but pushing the platform specific stuff into macros and functions and such, with some conditional compilation

That's got to be pushing the available ram considering we're already getting up there.

The big thing (one I was trying to solve when my day job took over and reduce my hobby hours) was to remove the C++ Arduino libraries from the compilation chain. C++ is nice, of course, but it also eats up code space faster. Still, the point is valid. a similar project for another platform that handles images and disk access to SD cards using FAT quickly moved beyond 32kB, but has stayed well under 60K, so 64kB could be a valid target. (4kB for a bootloader to update the app) And, RAM tends to increase as ROM does within a uC family.

But for instance the way pdduino handles the checksums kind of permeates everything because the "write-to-serial-port" function adds to the checksum for every byte passing through the function, and then because that is not actually right there has to be special exceptions here & there to send bytes that aren't part of a checksummed range. And now I don't want to do it that way.

I'm not sure I have a horse in the race, but I didn't think the checksum functionality was all that badly implemented. Yes, the special cases where the checksum is not to be used is a bit of a bother, but I think I factored all of that into something sane.

Then again, I may be in the wrong when it comes to targeting this environment. That checksum-as-you-go method was made by the original SD2TPDD author Jimmy Petit, and another thing he had originally which is no longer in here is he used a ring buffer for most everything, and it was kind of complicated keeping track of how it worked so me and Jim Brain replaced that with ordinary variables and buffers that get cleared and start from zero each time. Maybe the original way was actually the more proper embedded target way to do things because maybe it uses less ram and less cpu, for instance you don't really have to zero the buffer necessarily, just always move forward and just remember your current start and stop addresses, and zero any individual bytes that need it as you go, and basically just "be careful". So maybe his approach to the checksum was better too for this kind of environment. Maybe not universally I mean but maybe it was a case of doing the math that if most bytes do need the checksum (they do, by far), then treating the non-checksum bytes as the exception may use less cpu all around, even if it's a less convenient mental model.

I don't think you are wrong. the ring buffer took more housekeeping (you have to update the tail all the time, which is more work than just zeroing out the head pointer at the end. It also required more math (finding the opcode in a ring buffer means always calculating X bytes in front of tail, and creates lots of opportunity for "off-by-1" errors. Having the start of the packet begin at zero reduces those calcs to #defines, which can be pre-calculated and handles more quickly by the compiler/uC Ring buffers would have made more sense on the PC target (more abstraction, and you could potentially do memory mapped IO to the SD card or file).

Never the less, I will probably try to do it just like dlplus rather than like original SD2TPDD if I can get away with it, even if it might not be the leanest possible way, it will be the easier-to-get-right way. Which should be fine because it's like half way there now already. And maybe it's not really any leaner anyway.

[bkw777]

You also need the same checksum routine for verifying things you receive.

I suppose that could be done check-as-you-go also in a receiver function. And I suppose it could be made reasonably convenient by just having the rx and tx functions have a parameter, or maybe a global state bool that tracks what part of a packet we are in. But then also in the future we may want non-tpdd traffic like modem/wifi commands or who knows what other stuff, so, I keep coming back to seperation of concerns. It just seems to make the most sense that the checksum is a seperate function you use when you need it, not baked into something else that just happens to be the main consumer.

I think the cpu cost is the same. If there is any actual ram or cpu difference it's probably just in the fact that to do a checksum on a range all at once, it means you have to prepare the whole buffer in memory that will be summed, while to calc-as-you-go, you just need to know when to start and when to stop summing and don't actually need to have the whole buffer in memory, just the byte you're currently sending/receiving. But in reality of course we always have the whole buffer in memory anyways because it's small and way more convenient, plus, for recieving, you can't write to disk as you go anyway because you can't to do anything until you get to the end and verify the checksum.

This is not actually this important to justify this much discussion, I was just trying to imagine the theoretical most old-school, byte-counting, Wozniac way to do something just for the academic fun of it. Plus, the actual drive only has something like 2k of rom I think. So... there's that just sitting there mocking your best efforts haha

[flimshaw]

👋 OK got to tinker a bit more here and there today.

1. My mistake, the 8ms boost did seem to solve the loader issue, and I get the full message about TMP.DO etc. So mark that solved.
2. It turned out that I was not able to save files! I could reliably copy them to RAM, but saving to the PDDuino silently failed. I eventually modified the "O_WRITE" to "FILE_WRITE", just on a whim since that was a "recent" change, and that actually got me a bit further, though only creating empty files on the SD card with no errors. I can make / kill directories, and rename and kill files, but files I saved to the SD card show up empty. Copying takes longer based on file size, however.
3. Finally, I was getting frustrated so I wired up my serial cable and set debug 3 to try and see what was going on and, amazingly, files now save reliably. It only works if I replace the O_'s with FILE_s, and only when I am debugging to Serial. I immediately figured it was the minor delay the Serial log causes was fixing a timing issue, so tried a quick macro swap with some delayMicroseconds to match, but no luck. Writing only works when I'm debugging to serial.

Any ideas? haha. Here's a verbose log of a failed attempted file save, before swapping the O_ and FILE_ 102-failed-do-save-log-3.txt

And here's a successful log after doing so: 102-success-save-file-3.txt

(Updated on above: on second glance, different files and maybe different log levels. i'm fiddling too many knobs at once I think. lmk if you want cleaner/different logs)

Did a bit more digging and found the difference in SdFat:

This is O_WRITE: ./libraries/SdFat/src/common/FsApiConstants.h:#define O_WRONLY 0X01 ///< Open for writing only. ./libraries/SdFat/src/common/FsApiConstants.h:#define O_WRITE O_WRONLY

Vs. FILE_WRITE: ./libraries/SdFat/src/common/ArduinoFiles.h:#define FILE_WRITE (O_RDWR | O_CREAT | O_AT_END)

So maybe it's actually lacking some permissions it needs based on that change? But who knows why it only writes data if it's also DEBUG logging if it's not timing.

Thanks for indulging me above! I'm mostly following along. It would be nice to be able to use the TS-DOS wedge stuff like direct loading in basic etc. And it's true, if RAM turns out to be an issue, there are lots of options with a lot more.

[flimshaw]

Oh and if you are building successfully on an M0, could you share the arduino version you're using? Thanks again.

[bkw777]

You don't want FILE_WRITE, except you could use it for the OPEN_APPEND case. FILE_READ was ok, but if write can't be FILE_WRITE then it makes everything clearer to just use the same explicit syntax for all 3 modes. That O_AT_END just built in is really unexpected and non-intuitive to me. And in general, hidden implicit behavior causes exactly this problem where I used the wrong thing in the first place. This code is definitely tested after this point, but on 32u4 mostly. I don't remember when is the last time I actually used the m0 or teensy, but for that bit, it doesn't matter the platform.

[bkw777]

I do have a feather m0 mounted on a MounT sitting on my desk right now but I really want to figure out a config that doesn't use Adafruit's M0 library. I just don't trust it any more. Even when I thought I had figured out how to fix their spi timing error by overriding a value, I think it didn't actually fix it all the way and I didn't bother going back to update the support/forum posts where I said I though I had worked around it.

However I think the hardware is probably ok and maybe would be more reliable using just standard arduino m0/samd libraries. I don't know. Things are just weird on that device. Se the note about having to send a 2nd CR at the end of sendLoader() ? That is exactly true, and it does not happen with the same exact code on 32u4, or teensy 3.5 or teensy3.6. 32u4 is smaller and slower and 8bit, teensy is a totally different chip, 32bit, stupid faster (and also downclockable to 2mhz which is slower even than 32u4), and all thos different configurations work fine. Only feather m0 has this.

It's a shame because feather m0 would be nice. More horsepower to play with than 32u4, and it's one of the only 2 Feather models that has a sdcard reader built in.

Feather is really perfect for this because the assymetrical pin headers makes polarity protection, and the built-in lipo charger and the built-in card slot just makes it all a slam dunk. So it would be great if there wasn't just the single available ancient model to choose from. You can still buy adalogger 32u4 but man I can't believe that can continue for much longer right? m0 is really old by now too but it's at least a little newer.

Other than that, the next thing I want to do is get it working on one of the new models that has usb-c, wifi, and no card reader but does have 4 or 8 M flash built-in and has some kind of built in easy "emulate a mass storage" so that although you don't have an sd card, you can just plug the usb cable in and access the files the same way. 8M or even just 4M is actually quite a lot for TPDD files, so I think that would actually make a perfectly good usable TPDD, and the we have the wifi and the horsepower to try adding in zimodem.

An older one with 4M and 8266 and microusb https://www.adafruit.com/product/3046 A newer one with 8M, esp32 usbc https://www.adafruit.com/product/5400 But it's esp platform which I have not really managed to use successfully yet.

The teensy's both work great, but, symmetric pin headers means it's easy for the user to plug in backwards, and no lipo manager. Also the teensy's card slots don't have anything connected to the card detect switch, so you can't have an interrupt monitor the card slot the way we do for Feather.

[bkw777]

I never verified, how are you connecting the feather to the 100? You need a max3232 in there in some form. You shouldn't need any manual button to fake dsr because it should be connected through the max3232. If you're not using a MounT, then you have to cobble up some other ttl-rs232 adapter that has more than just rx/tx and allows you to rearrange the connections so you can use the 2nd channel for dsr/dtr instead of rts/cts, or just use a max3232 breakout board.

[bkw777]

I do not know what's up with this thing. I just tried both my 32u4 and my M0 and both built and uploaded ok, and 32u4 works fine and M0 is just broken. I used both the arduino 2.0.0-rc9.2 beta and 1.8.19

I guess I have to back up and sart over with this thing, just make a light blink, test each pin input and output with a simple button and led and establish that I can even operate the device at all.

Maybe I even just have a simple bad connection on one of the pins. dtr/dsr detection is working, the sdcard is working, sleep is not working so I disabled that too, and the best I can get is I can see the sdcard file list on the console tty with debug enabled, and issue commands with pdd.sh and see the commands appear, but not see anything coming back to the client.

[bkw777]

Might want to try this old version, it's the last version after adding M0 support, before adding sleep for M0 https://github.com/bkw777/PDDuino/tree/d35d8a7cf7af0b0fda4d3ac13609ea40ddca8c2f And then the next commit adds M0 sleep (which I think worked at the time) Just to see what happens. There's a note in the readme about modifying a gcc option buried in an ide file. We don't need to do that any more on the current code, but I think that old code was wrong in some way that compiler didn't like and that -fpermissive is needed to let it build. So, you can either do that edit like the readme says, or figure out what the compiler doesn't like and fix the code. This is really old and for instance doesn't even have sendLoader() yet. But it might show something.

Also, might not be so hard to try building with the arduino instead of the adafruit samd board support:

  // only needed if you don't have the Adafruit SAMD board support installed
  //#if defined(ARDUINO_SAMD_ZERO) && defined(SERIAL_PORT_USBVIRTUAL)
  // // Required for Serial on Zero based boards
  // #define Serial SERIAL_PORT_USBVIRTUAL
  //#endif

Well if that's all it takes...

[flimshaw]

I never verified, how are you connecting the feather to the 100? You need a max3232 in there in some form. You shouldn't need any manual button to fake dsr because it should be connected through the max3232. If you're not using a MounT, then you have to cobble up some other ttl-rs232 adapter that has more than just rx/tx and allows you to rearrange the connections so you can use the 2nd channel for dsr/dtr instead of rts/cts, or just use a max3232 breakout board.

Ah that might be helpful! So my setup is the following shoved/epoxied into a 3d printed enclosure:

1. adafruit feather m0 adalogger + 500mah LiPo
2. no-name 4-pin TTL->RS-232 adapter (I believe this one). also I have jumpered the DTR-DSR-DCD pins together per your README
3. DB-9 to DB-25 mini null modem adapter (something like this one). this jacks into the back of the Model 100
4. I've got a 10k resistor to ground through a momentary switch on pin 6, if i press that while plugging in the SD card, it sends the loader
5. I also broke the SD activity out to an external LED on pin 9

Only photo I have handy is while it was breadboarded:

in this case, i'd hard wired the 10k to gnd to always have it bootstrap ts-dos

[flimshaw]

100% agree on all your points about the feather, it really does have all the pieces sitting there for you, power/charging included, minus the serial connector. A little FeatherWing that adds a Max3232 and a DB-25 would be pretty trivial to whip up from OSH park or whatever, but also a 5 pack of these serial adapters are pretty cheap and easy to wire.

And strange about those bugs w/ Adafruit's libraries. Ever try and start a thread on their forums? I wonder if they'd have any ideas, might be willing to help track it down. Would be so nice to get this working with ~$30 worth of parts you can pick up at microcenter and a little soldering.

My setup has been working quite well since swapping all the O_ with FILE_, as long as I have at least DEBUG 1 enabled. (I don't fully grok all the SD perms you outline above, but these are the only settings that i've gotten working here). I commented out the code to wait for the Serial adapter at the beginning just to keep it debugging when not connected over USB, and I haven't had any errors on either machine since 👍 👍 👍 .

In order for it to work reliably I always do it in the following order:

1. turn off M100
2. connect/turn on PDDuino, wait for SD flash
3. turn on M100, launch TS-DOS etc

[bkw777]

"A little FeatherWing that adds a Max3232 and a DB-25 would be pretty trivial to whip up from OSH park" https://github.com/bkw777/MounT

[bkw777]

FILE_WRITE is no question wrong on the OPEN_WRITE line. It's ok on the OPEN_APPEND line.

There are seperate TPDD commands to open a file and write to it on overwrite vs append modes, and when a client sends the command to open in write mode, it expects to be writing or overwriting the file from the beginning.

FILE_WRITE is a macro containing a value that represents bit flags, and includes the O_APPEND bit flag, which you can not have when the client has not explicitly asked for it.

The only reason it seems to work is because when you create a new file and save it once, appending to the end of a new zero-byte file is the same as starting from the beginning of a new zero-byte file. Possibly even re-saving an existing file still works if you're just lucky and the client happens to be written to issue commands to delete the file before saving. I don't know off-hand what TS-DOS does, but for instance teeny will definitely not be doing any extra steps like that, nor will any application that wants to use a file for data.

FILE_READ and FILE_WRITE are just one of those many Arduino oversimplifications that makes something actually wrong or at least obscures what's really going on, and obscures how to do something with more control by not showing any other way to do it. The Arduino manual page for open() just says those are the only 2 modes and doesn't show anything else, but "open for reading" and "open for reading and writing starting at the end of the file" are not the only two options, and you do need more than those two options. I guess they expect you to do a seperate seek() if you need to do anything other than append? At least they do provide a seek()!

Actually, it seems even worse than that.. it's not only wrong it's inconsistent and you never actually know what it's going to do. Apparently not all board support libraries SD wrappers have the O_APPEND included in FILE_WRITE, and the main Arduino SD support has had it, removed it, then added it back at different times. https://arduino.stackexchange.com/questions/65888/arduino-sd-card-open-file-modes-append-overwrite

[bkw777]

I went back and looked at the original SD2TPDD, and it gets even kookier. In his case, working with a regular Arduino mega, and a years-ago version of Arduino, apparently in that case FILEWRITE did the opposite and didn't offer any way to append. (it did of course, because you just don't have to use the FILE* macros at all. It's just crappy docs that lies.)

So he ended up doing this:

bool append = false;  //SD library lacks an append mode, keep track of it with a flag
[...]
          case 0x01: entry = SD.open(directory, FILE_WRITE); append=false; break; //Write
          case 0x02: entry = SD.open(directory, FILE_WRITE); append=true; break;  //Append, set the append flag
          case 0x03: entry = SD.open(directory, FILE_READ); append=false; break;  //Read
[...]
    if(append){
      entry.print(dataBuffer[(byte)(tail+4+i)]);  //If the append flag is set, use "print" to append to the file instead of "write"
    }else{
      entry.write(dataBuffer[(byte)(tail+4+i)]);
    }

[bkw777]

Ok enough with the theory & guessing...

  // info dump
  DEBUG_PRINTL(F("\r\n-----------[ " SKETCH_NAME " " SKETCH_VERSION " ]------------"));
  DEBUG_PRINTL(F("BOARD_NAME: " BOARD_NAME));

  DEBUG_PRINT(F("O_READ: "));
  DEBUG_PRINTIL(O_READ,BIN);
  DEBUG_PRINT(F("O_WRITE: "));
  DEBUG_PRINTIL(O_WRITE,BIN);
  DEBUG_PRINT(F("O_APPEND: "));
  DEBUG_PRINTIL(O_APPEND,BIN);
  DEBUG_PRINT(F("O_SYNC: "));
  DEBUG_PRINTIL(O_SYNC,BIN);
  DEBUG_PRINT(F("O_CREAT: "));
  DEBUG_PRINTIL(O_CREAT,BIN);
  DEBUG_PRINT(F("O_EXCL: "));
  DEBUG_PRINTIL(O_EXCL,BIN);
  DEBUG_PRINT(F("O_TRUNC: "));
  DEBUG_PRINTIL(O_TRUNC,BIN);
  DEBUG_PRINT(F("FILE_WRITE: "));
  DEBUG_PRINTIL(FILE_WRITE,BIN);

Compiled and ran on both devices and got this. The different lengths within one platform are the print function doesn't zero-pad so leading 0's are missing. The different lengths between the two platforms are 32u4 is 8bit abd M0 is 32bit So "1" is really "00000001" on 32u4 and "00000000000000000000000000000001" on M0 I manually right-justified to make it easier to match up the bits.

BOARD_NAME: Adafruit Feather 32u4
O_READ:                   0
O_WRITE:                  1
O_APPEND:              1000
O_SYNC:            10000000
O_CREAT:              10000
O_EXCL:             1000000
O_TRUNC:             100000
FILE_WRITE:           10110

BOARD_NAME: Adafruit Feather M0
O_READ:                   0
O_WRITE:                  1
O_APPEND:              1000
O_SYNC:      10000000000000
O_CREAT:         1000000000
O_EXCL:        100000000000
O_TRUNC:        10000000000
FILE_WRITE: 100001000000010

And I'm confused by the results, especially when looking at https://github.com/arduino-libraries/SD/blob/master/src/utility/SdFat.h Where it looks like O_READ should be 1 not 0 ?

As far as I can tell, the two different platforms, on the same day, using the same version of Arduino, have two different meanings for FILE_WRITE, but it appears that neither one includes the O_APPEND bit. Yet, the user-facing manual does very clearly promise "starting at the end" https://www.arduino.cc/reference/en/libraries/sd/open/

This is a garbage interface that should not be used. Might as well say mode=random.

[bkw777]

Oh my goodness am I kicking myself... I just pushed this update

          case F_OPEN_WRITE: entry = SD.open(directory, O_WRITE | O_CREAT | O_TRUNC); break;
          case F_OPEN_APPEND: entry = SD.open(directory, O_WRITE | O_APPEND); break;
          case F_OPEN_READ: entry = SD.open(directory, O_READ); break;

F_OPEN_WRITE was missing O_CREAT

One question still is if append should include create.

The tpdd manual says that append mode is only for existing files, so that's what this does. But I'd really have to use pdd.sh to send some manual commands to test what a real drive actually does if you try to open a file in append mode that doesn't yet exist.

Same goes for the O_TRUNC above. The manual says that write mode is for new files, so, whether that means it truncates an existing file or just starts writing from the beginning, I'd have to do an actual test to see what a real drive does, but I'm pretty sure it should truncate.

So that explains your experience. That commit that removed FILE_WRITE did break it, and FILE_WRITE is still wrong but it's better than what I had in there.

[flimshaw]

👋 Thanks for all the above, finally caught up here.

I just flashed your latest to my board, still had to disable sleep, but all file writing/reading/bootstrapping working well. I think that is making more sense given my setup though, should the processor still wake up when the M0 only has TX/RX and no other pins?

Anyhow, I ordered a batch of your feather MounT boards from OSH, so I won't spend too much time worrying about this prototype. It's definitely too gnarly looking to take to the coffee shop anyhow 😅

I also got a usb/rs232 cable and gave dlplus a brief go, also working well, including direct loading of BASIC files from disk with the "0:" syntax. At this point, it's already so useful to just have a reliable way to bootstrap, save/load docs/software off of SD. I'm not missing the "0:" bells and whistles too much, but there's probably software and use cases I'm not aware of right?

I also put an order in for a Rex#, since it seems awfully easy to overwrite TS-DOS when it's in RAM, but no ETA on those at the moment.

[bkw777]

Well, If you're adventurous, another option is you can build your own REX Classic. http://tandy.wiki/REX

As for sleep, a few things:

I can't say for sure it works for M0, but at least on the others, all it takes to wake up is for the client to send anything and create activity on the RX pin.

The code currently only looks at the RX pin and the card detect pin for interrupts. Maybe the uc has other interrupts internally, any interrupt wakes it from sleep, but the code only sets up those 2.

But it requires that the uc allows setting an interrupt on that pin.

Another thing is maybe the deepest sleep is too deep but there are other levels that might still work and would still be better than nothing. I don't remember if I have a configurable #define or just some comments in sleepNow(). Maybe try going up one level from the deepest.

Another thing is you can't have debug enabled at the same time as sleep. The only way to debug sleeping is with the led.

Another thing is the 32u4 and M0 are using two different libraries for sleeping. Even though 32u4 is working, the M0 version of the sleep code could just be borked since it's a different .h file and different functions and I haven't worked on it or tested it as much. It did work originally but maybe an update changed something, or I did at some point.

Another thing is maybe the M0 just needs different handling, like, have a really long sleep delay so that it doesn't try to sleep instantly but stays awake until it's been idle for a few minutes, then you just do an extra dir list to wake it up when you want to use it, sometthing like that.

You shouldn't have to do that normally but I'm just saying possible work arounds and next-bests etc until it's figured out.

-- bkw

On Tue, Aug 30, 2022, 8:33 AM Charlie Hoey @.***> wrote:

👋 Thanks for all the above, finally caught up here.

I just flashed your latest to my board, still had to disable sleep, but all file writing/reading/bootstrapping working well. I think that is making more sense given my setup though, should the processor still wake up when the M0 only has TX/RX and no other pins?

Anyhow, I ordered a batch of your feather MounT boards from OSH, so I won't spend too much time worrying about this prototype. It's definitely too gnarly looking to take to the coffee shop anyhow 😅

I also got a usb/rs232 cable and gave dlplus a brief go, also working well, including direct loading of BASIC files from disk with the "0:" syntax. At this point, it's already so useful to just have a reliable way to bootstrap, save/load docs/software off of SD. I'm not missing the "0:" bells and whistles too much, but there's probably software and use cases I'm not aware of right?

I also put an order in for a Rex#, since it seems awfully easy to overwrite TS-DOS when it's in RAM, but no ETA on those at the moment.

— Reply to this email directly, view it on GitHub https://github.com/bkw777/PDDuino/issues/5#issuecomment-1231604628, or unsubscribe https://github.com/notifications/unsubscribe-auth/AM7PONNSFTS2RGNZMHU42I3V3X5TLANCNFSM57NCZW4Q . You are receiving this because you commented.Message ID: @.***>

[bkw777]

It's easy to overwrite any machine language in ram on the 100 because of how the cpu has no relative-jump instruction, machine-language programs aren't position-independent. They get compiled to run at a certain address in memory, maybe with an option to change that one time at install-time, and then that's it, That range of memory addresses is unavailable for any other use as long as the program ist installed. It can't move, and if some other program wants the same area, it clobbers the first program.

The normal way around that is the .CO file, which has a header that says what the begin/end/exec addresses are for that program. You can store a .CO file in the ram filesystem anywhere without knowing it's exact location, and have a small .BA one-liner that clears the necessary high memory area and launches the .CO file each time you want to run it. That makes the .CO file work more or less like in a modern OS where you just run things without worrying about actual memory addresses.

But it's very ram-wasteful. TS-DOS is something like 5 or 6k. To use the ram version in the form of a .co and .ba launcher, it means there is a copy of the program in some random address in the filesystem area, and each time it's executed there is another copy in the actual address it was compiled to run at in high memory. for a 5 or 6k program it means it consumes 10 or 12k of ram, when you only have about 29k available just turning the machine on before doing anything else.

To save precious ram, it's possible to install the actual execution copy alone, and not have a 2nd copy in the form of a .co file, and just have a menu entry file that's only a few bytes and all it does is jump to the execution address. That's called a trigger file. For this to work more than one time, you have to very carefully never allow any other program to overwrite or corrupt the high memory address range where the program is.

Many programs do provide an installer/loader that can modify the machine language one time at install-time, so that multiple machine language programs can be stacked up in high memory instead of clobbering each other, but it's tricky and fragile.

So for the most convenient way to operate day to day is to voluntarily limit yourself to using an option rom for as much as possible, and then pick just one or a very few machine-language apps to be installed in ram and you just avoid running any other random machine language apps so that your high memory area doesn't get clobbered.

You want to have your largest and/or most-used machine-language app in the form of an option rom for starters. Option rom code sidesteps all of this. But, without a REX, you can only have a single option rom installed at any given time, which is why there were so many multi-function option roms that tried to provide a whole suite of apps in the same rom.

This is also why Ultimate ROM II has on-the-fly loaders for TS-DOS and Sardine. It was actually worth it. The rom could be 99% devoted to application code, and had just enough TPDD code to load TS-DOS from disk, execute it, and throw it away.

Normally TS-DOS would be something you use so often, and it's large enough to be annoying to keep in ram, that it's a perfect candidate for wanting it in the form of an option rom instead of ram (that also makes it soooo much more convenient to re-install after a hard reset). But if you want a fancy office suite, that doesn't leave enough room for both an office suite and a full-fat tpdd client like ts-dos both in the 32k available for option rom. But since the whole point of TS-DOS is just to access a disk, it makes it possible to store TS-DOS on the disk instead of on the machine, and it's not that much of an extra hassle, because generally the only time you need to use ts-dos is when you have your disk drive. Just make sure every disk contains a copy of DOS100.CO, or at least make sure at least one disk is always with you that has it.

Anyway, so you have your biggest and/or most-used application in option-rom, now you can still have one or a few machine-language apps installed in ram too, and just limit yourself to one or a few so that you just install them once right into their execution address with no 2nd copy in the form of .co files, and don't run any other random .co files so that your one or few apps you really want don't get clobbered.

REX is just the best. With that, you can have both the full standard rom version of ts-dos which uses no ram and conflicts with no other machine language apps, and still have all the other option roms like Lucid etc, and not have to bootstrap to recover from a hard reset, plus just as a bonus, multiple full ram image backups, right on-device, recoverable in 2 seconds from a hard-reset any time with nothing else needed (no disk or drive etc, totally self contained).

[go4retro]

On 8/26/2022 4:43 PM, Brian K. White wrote:

I went back and looked at the original SD2TPDD, and it gets even kookier. In his case, working with a regular Arduino mega, and a years-ago version of Arduino, apparently in that case FILEWRITE did the opposite and didn't offer any way to append. (it did of course, because you just don't have to use the FILE* macros at all. It's just crappy docs that lies.)

So he ended up doing this:

|bool append = false; //SD library lacks an append mode, keep track of it with a flag [...] case 0x01: entry = SD.open(directory, FILE_WRITE); append=false; break; //Write case 0x02: entry = SD.open(directory, FILE_WRITE); append=true; break; //Append, set the append flag case 0x03: entry = SD.open(directory, FILE_READ); append=false; break; //Read [...] if(append){ entry.print(dataBuffer[(byte)(tail+4+i)]); //If the append flag is set, use "print" to append to the file instead of "write" }else{ entry.write(dataBuffer[(byte)(tail+4+i)]); } |

— Reply to this email directly, view it on GitHub https://github.com/bkw777/PDDuino/issues/5#issuecomment-1228972721, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGXYKSIEWNWRDFKYYYD7OHLV3E27PANCNFSM57NCZW4Q. You are receiving this because you commented.Message ID: @.***>

This is exactly why I was trying to replace the SdFat library in the codebase with FatFS, which is smaller, does not require C++, uses less RAM, and has very granular options in this space.

Jim

-- RETRO Innovations, Contemporary Gear for Classic Systems www.go4retro.com store.go4retro.com

[go4retro]

On 8/25/2022 5:36 PM, Brian K. White wrote:

You also need the same checksum routine for verifying things you receive.

Yep.

I suppose that could be done check-as-you-go also in a receiver function. And I suppose it could be made reasonably convenient by just having the rx and tx functions have a parameter, or maybe a global state bool that tracks what part of a packet we are in. But then also in the future we may want non-tpdd traffic like modem/wifi commands or who knows what other stuff, so, I keep coming back to seperation of concerns. It just seems to make the most sense that the checksum is a seperate function you use when you need it, not baked into something else that just happens to be the main consumer.

I see where you are going, and I think there's no concerns.  I still believe discarding the ring buffers is a sound design principle, but in actuality, your idea about calculating the checksum at the end is more "embedded OS friendly".  The idea is that you don't do anymore than you need to do when you are receiving events (RS232), as you might lose events if you do too much.  Then, when you are at a logical stopping point, you take the time you need to do the math.

And the separation of concerns is a valid argument here, as you note.

I think the cpu cost is the same.

It's mostly a wash on embedded CPU architectures.  The inline calc takes a global var, requiring a pull from RAM and push to RAM from a register.  Separate function can just use a register, no global var to push or pull. Assuming the branch for the loop doesn't introduce a penalty, the only thing the looping function has to do extra is the loop index math.  Assuming all those are 1 cycle issues, you either have 2 extra cycles (push and pull) for the pay as you go method per byte, or 2 cycles (increment index and branch) per byte for the looping function. The looping function loses a few cycles for the extra call to the function itself, but that's noise, and you dispense with the global var.  A wash. So... there's that just sitting there mocking your best efforts haha

Indeed.

Jim