Suggestion(?): File transfers to/from SD card?

[MarkJelic]

NOTE: I have only rudimentary coding knowledge, and have next to zero knowledge of how the Pi works. So if this Suggestion is dumb - I apologise in advance.

So I'm having a discussion with some people on the RC2014 Facebook group and someone brought up that the reason they don't use PiGFX is because it doesn't have any file transfer functionality. (I'm taking that as being truthful. Please correct me if it can indeed do them.) Which I agreed with him and got me thinking:

The Pi Zero has a SD card. The PiGFX part of that software takes up less than 1gig of space leaving heaps left over. Why can't we make use of it? Could we make some extended escape codes that 1) List the files that are on the SD card and assign a numeric tag to each. 2) Initiate a file download using the numeric tags from above (of course, your Monitor program would then have to be able to accept and do something with those files.) 3) With a third new escape code, it saves to a file that you can name. (And again, the Monitor would need to be engaged with the file transfer.)

Thoughts? Put it on the wishlist or is this an unworkable idea?

[chregu82]

Ok, here are my thoughts. At the moment there's no such feature in PiGFX. It's able to read data from the SD card, write access is not implemented. This would have to be done. The next thing would be finding a useful way to interact with the RC2014 or any old computer with a serial interface. What would be expected? Sending escape codes to PiGFX which then responds with a binary data stream of the file? Or should we somehow emulate a harddrive which would be accessible within CP/M? Or would it be useful to just load a file from the SD card into RAM and run it? Next problem probably is, that the Pi would probably overload the serial input buffer if file data is sent.

[MarkJelic]

Good questions and thank for the reply, because you bring up some good points. I'll be honest and say I have not used the RC2014 much as I only pretty much built it a week or so ago. And tbh I don't have a lot of experience with serial terminals, either... But I get all the concepts.

I watched this video today... I don't own the z80ctril board but seems like it would be a good one. Here is an overview video: https://www.youtube.com/watch?v=sVV1PYWX18A&t=637s

At about the 3:20 mark, he starts talking about file interactions. Could PiGFX do similar?

[chregu82]

I think I understand how this file transfer works. You put the RC2014 into some kind of listening mode and initiate the file transfer on the sending system. The problem is, how would you tell the Pi what to send, after you put the RC2014 into this listening mode?

[BleuLlama]

I started to work on a "vampire-tap-like" drive interface bit of hardware for the RC2014, thinking that I would eventually add the functionality into PIGFX... basically it would do (x|y|z)-modem like transfers, to an sd card over the same serial connection as the console. I had initially designed it to plug right into the FTDI header on the RC2014, but the functionality could also easily be implemented in PIGFX, or a terminal application on a host system... I had started with an extension/modification to miniterm.py.

Like many of my other projects, I wandered off to another shinyshiny before finishing it... perhaps it's time to revisit it?

I know it would be slow as hell, but it would never be as slow as a Commodore 64, and hey, it'd be free, since it wouldn't specifically require additional hardware to implement.

The base feature set i was working towards:

• Escape-sequence based interface that wouldn't interfere with standard VT100/ANSI escape sequences
• Capture text to a file - for logging or saving stock-RC2014 BASIC programs
• "type" text from a file - perfect for "loading" programs into stock RC2014 BASIC, or typing in hex files to the loader.
• random file io. (like the CH376 interface chip's command set, but sane.)

Then:

• subdirectory support

Eventually;

• commands for CP/M to do normal file transfer/utility stuff. Copy files to/from SD card.
• extensions to add additional virtual files/protocols that can be remote devices... ie http:127.0.0.1/readme.txt or telnet://129.21.60.4:1701
• CP/M BIOS to natively use this interface instead of the expensive CF interface... perfect to try out CP/M without paying anything extra. ;)

so... should I return to this?

-s

On Sun, Nov 22, 2020 at 2:27 AM Christian Lehner notifications@github.com wrote:

I think I understand how this file transfer works. You put the RC2014 into some kind of listening mode and initiate the file transfer on the sending system. The problem is, how would you tell the Pi what to send, after you put the RC2014 into this listening mode?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/fbergama/pigfx/issues/49#issuecomment-731709865, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAQ7GGP5MOCHV47TBBNBN3SRC4MTANCNFSM4T3H2QDA .

-- Scott Lawrence yorgle@gmail.com

[chregu82]

I think this sounds absolutely cool and would give PiGFX a great new set of features.

[MarkJelic]

You put the RC2014 into some kind of listening mode and initiate the file transfer on the sending system.

As I said I've only just put the RC2014 system together and today I got to play with serial transfers. The Monitor it comes with (the Small Computer Monitor) seems to be looking at the input stream and if the input text is in an Intel Hex format, it does the usual thing of loading it into the defined memory location. So from the Terminal software, depending on what you run, you either initiate a "Send File" command, or in the case of TinyTerm, you copy the Intel Hex file to the clipboard and then Right Click > Paste it onto the window ans the SCM does its thing.

The problem is, how would you tell the Pi what to send, after you put the RC2014 into this listening mode?

Well, that could be done a couple ways; As per my OP, one of the commands could be doing a DIR command that gets streamed back to the computer as if it was typed by the user. Also, in terms of telling the Pi what to send, sending escape codes that also have the file name I think would cut it, no?

[MarkJelic]

Like many of my other projects, I wandered off to another shinyshiny before finishing it... perhaps it's time to revisit it?

Yeah, I think we all suffer from that. :) I think it would be great if you could revisit, even just starting with a very simple "Write to a text file", and "read in a text file".

I know it would be slow as hell, but it would never be as slow as a Commodore 64, and hey, it'd be free, since it wouldn't specifically require additional hardware to implement.

Exactly. And I'm sure it wouldn't be that slow to make anyone care.

The base feature set i was working towards: - Escape-sequence based interface that wouldn't interfere with standard VT100/ANSI escape sequences -

Perfect!

Capture text to a file - for logging or saving stock-RC2014 BASIC programs

I think this is going to be the trickiest part, as to output onto screen being "caught" by the Pi and sending instead to a file... That I think will require some modification to the SCM. For example, the command to Disassemble would ideally have an option of a PIPE "|" that has the escape codes required to start the Pi saving to a file. For example

• D 8000 | Example.asm This would print to screen the normal disassembler text, but before doing so it would send the escape codes for Pi to grad and create a text file called Example.asm And then we could extend SCM to output the contents of RAM as an Intel file, again, starting with the hidden escape codes for the Pi.

"type" text from a file - perfect for "loading" programs into stock RC2014 BASIC, or typing in hex files to the loader.

Yep, perfect. I honestly think this is the easiest part to implement, as the SCM already knows how to handle incoming Intel Hex files. See if you can quickly implement this to see if this concept is viable.

random file io. (like the CH376 interface chip's command set, but sane.)

Not familiar with this, but sure, put it on the wishlist. :)

Then: - subdirectory support

As in, creating and deleting folders, or do you mean just allowing subdirectory names to be used as part of the filename?

Eventually; - commands for CP/M to do normal file transfer/utility stuff. Copy files to/from SD card.

Nice.

CP/M BIOS to natively use this interface instead of the expensive CF interface... perfect to try out CP/M without paying anything extra. ;)

Totally agree, because who the hell uses CF cards any more?? It's not like they were around in the 80s, and they certainly aren't in wide use today... So it matches no era and we may as well get with the times. :)

so... should I return to this?

Ummm... YESSS! lol

So when can we expect a beta? ;)

Thanks for your input and look forward to anything you want tested. :)

[BleuLlama]

Heh. Thanks. (fwiw. i responded to this thinking it was an email thread, since i'm responding via mail... it didn't click until just now that this was a thread on one of pigfx's github issues. Apologies for the hijack. I'll post more to the RC2014 email list about these bits shortly. :)

On Mon, Nov 23, 2020 at 6:37 AM Mark Jelic notifications@github.com wrote:

Like many of my other projects, I wandered off to another shinyshiny before finishing it... perhaps it's time to revisit it?

Yeah, I think we all suffer from that. :) I think it would be great if you could revisit, even just starting with a very simple "Write to a text file", and "read in a text file".

I know it would be slow as hell, but it would never be as slow as a Commodore 64, and hey, it'd be free, since it wouldn't specifically require additional hardware to implement.

Exactly. And I'm sure it wouldn't be that slow to make anyone care.

The base feature set i was working towards: - Escape-sequence based interface that wouldn't interfere with standard VT100/ANSI escape sequences

Perfect!

Capture text to a file - for logging or saving stock-RC2014 BASIC programs

I think this is going to be the trickiest part, as to output onto screen being "caught" by the Pi and sending instead to a file... That I think will require some modification to the SCM. For example, the command to Disassemble would ideally have an option of a PIPE "|" that has the escape codes required to start the Pi saving to a file. For example

• D 8000 | Example.asm This would print to screen the normal disassembler text, but before doing so it would send the escape codes for Pi to grad and create a text file called Example.asm And then we could extend SCM to output the contents of RAM as an Intel file, again, starting with the hidden escape codes for the Pi.

"type" text from a file - perfect for "loading" programs into stock RC2014 BASIC, or typing in hex files to the loader.

Yep, perfect. I honestly think this is the easiest part to implement, as the SCM already knows how to handle incoming Intel Hex files. See if you can quickly implement this to see if this concept is viable.

random file io. (like the CH376 interface chip's command set, but sane.)

Not familiar with this, but sure, put it on the wishlist. :)

Then: - subdirectory support

As in, creating and deleting folders, or do you mean just allowing subdirectory names to be used as part of the filename?

Eventually; - commands for CP/M to do normal file transfer/utility stuff. Copy files to/from SD card.

Nice.

CP/M BIOS to natively use this interface instead of the expensive CF interface... perfect to try out CP/M without paying anything extra. ;)

Totally agree, because who the hell uses CF cards any more?? It's not like they were around in the 80s, and they certainly aren't in wide use today... So it matches no era and we may as well get with the times. :)

so... should I return to this?

Ummm... YESSS! lol

So when can we expect a beta? ;)

Thanks for your input and look forward to anything you want tested. :)

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/fbergama/pigfx/issues/49#issuecomment-732106277, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAQ7GHQY5CSVEYTBC5X23TSRJCOHANCNFSM4T3H2QDA .

-- Scott Lawrence yorgle@gmail.com

[rkincaid]

I would probably use the file transfer functionality if it were available. I'm interested in connecting to a small 6502 SBC. It already has support for invoking client-side XMODEM transfers via BASIC SAVE and LOAD commands. This seems to be somewhat prevalent on the really low end 8-bit schemes folks have been coming up with. If the basic pgfx firmware had the ability to invoke and XMODEM send/receive from the SD card, it would be pretty handy (at least for me grin). Both ExtraPutty (windows) and minicom (Mac/Linux) have UI's for doing exactly this with a serial connection. So those would be good models for how to implement the UI/functionality. For extra credit, a full ZMODEM implementation would be cool, but XMODEM alone is probably simpler and sufficient. FWIW...

[chregu82]

I could probably quite easily implement something to send a file with XMODEM to the host. But I really don't know a good way for selecting the file. Any suggestions about that? As long as FAT write access is not implemented saving something to the SD card is not yet possible.

[rkincaid]

Yeah... I see now. I saw that uspi was recognizing the SD card as a device, but I realize now it only supports bulk transfers. It probably isn't too hard to hack some kind of very rudimentary storage structure to store and retrieve a few BASIC files on a dedicated partition at known locations, but it wouldn't be readable on a Mac/PC as a real file system. I'll think some more about this to see if there's an approach that might work or if there's any existing code that could be applied to the problem.

[chregu82]

Ok, I had some time to implement something. It's in the filetransfer branch. It's no much yet, but maybe worth a try. You'll have to create a directory "transfer" on the SD card. There are 2 new commands: ESC[!!d List files in "transfer" folder on the SD card ESC[!filename.ext!t Type file named "filename.ext" to serial port (ASCII)

[rkincaid]

Thanks. Great start. I'm waiting on some new SD cards, but will try this soon (I've already successfully compiled the branch). The only issue for my needs are that the sending the text to the serial port needs to have delays (ideally user specified in a config file somewhere) that has both an inter-character delay as well as an end-of-line delay. If you stream this into a home-brew SBC, chances are it won't have flow control and won't interpret the input stream fast enough to avoid dropping characters and getting errors. I've successfully worked around this with a number of terminal apps like TeraTerm and minimum using such delays (which they support). Don't feel the need to jump in and do this right away (unless you feel like it). I see enough in the source code to hard-code a few things for now and probably get it to work. Just as an example, I use a 10ms character delay and and 100ms line delay for my SBC and it works great at 9600 baud (the speed my SBC is hardcoded at). Also, to explain a bit, the inter-character delay is just to avoid overflowing the input buffer with no flow control. The line delay is to give the interpreter time to parse a line of basic and do its internal management of the tokenized BASIC code. This will probably vary a lot depending on the SBC, hence the need to be user-specified.

One other thought, but maybe breaks your design goals... minicom and screen (Linux/Mac terminal programs) invoke their non-display functions via a user-settable control character (usually ctrl-a). Using the Esc function makes sense to get started, but again is problematic for me as Esc from the keyboard is also used to stop any transfers. Not really a problem at the moment, but if we ever wanted to invoke xmodem, I would need to start a transfer on the SBC and then invoke the corresponding end in pigfx, which would probably cancel the SBC end. Again, not something to deal with yet, but maybe something to think about from a usability perspective.

Again, thanks for implementing this great beginning.

[chregu82]

I already had the feeling, that speed would be an issue. So I implemented 2 parameters in the config file for setting a character and a line delay, more or less like TerraTerm does. Xmodem is definitely one of my goals, so I'm going to have a look at your ctrl-a suggestion. This character could probably be used the same way as the escape character.

[chregu82]

I tried the type file feature with a RC2014 mini in BASIC. It seems to work. I also uploaded 2 samples into bin/transfer. In BASIC just type: 10 print chr$(27) + "[!quest.bas!t" or 10 print chr$(27) + "[!masterm.bas!t" and RUN

There will be no display output while data is transfered. It takes some time... After loading you should be able to list the program or run it. For starting a transfer you should choose a line number which gets overwritten by the data you are loading.

[rkincaid]

Ok, I finally got things working a bit. Just FYI I ran into an interesting problem... The new SD cards I ordered were class 10 (so I can use them for other things) and apparently pigfx doesn't like that. I kept getting a prefetch error on boot no matter what I tried. In desperation I went back to my older class 4 SD card and that worked fine. And yes I wasn't running the full 8 or 4Gb of the card. I was just running inside the original boot partition that's less than 1 GB. So apparently the class 4 works but the class 10 doesn't. Ugh. And the class 10's work fine as normal PiOS volumes.

At any rate I verified that everything works as advertised (including the transfers) on my pi just echoing via a bonded GPIO pins. I need to wire up a level shifter in order to hook to my SBC. That will take a few days for me to get the time. Stay tuned...

[rkincaid]

quick update... As a interim test I ran wires between two raspberry pi's so I didn't need level shifters. Running a version of BASIC one PI and pigfx on the other I was able to successfully transfer some programs. So seems to work. I'll give it some more effort later this week and maybe tinker with it some. Thanks again.

[rkincaid]

OK, verified working between pigfx and my 6502 SBC. Thanks. I'll tinker around with the code a bit. If I have any code changes to share what is your preferred way to pass those along?

[chregu82]

I don't know, post it here or commit it to your fork. I'm planing to change it again, so it doesn't pause everthing else while sending data.

[rkincaid]

So a few observations on the current build:

1. Filenames longer than 8.3 form get mangled to an 8.3 representation when listing the transfer directory. I noticed the code that reads the directory actually does this explicitly in fat.c?, so presumably it knows the full name but choses to go 8.3 for some reason. That wouldn't be so bad but it can't reverse engineer this to load the file from the SD card using the full name. I may try just commenting out the 8.3 conversion and see if that works. Oddly, even if I try to load the file using the full unmangled name that fails too.
2. Short files transfer fine, but longer files seem to hangup in some continuous transfer mode. The behavior on my SBC is that BASIC file transfers silently to the SBC and the the entire program dumps at the end (not from invoking a LIST command). It would be better to echo back any SBC characters as they are written to the SBC so you could see the actual progress. It's particularly important since if there are syntax errors, etc. Otherwise you won't be able to see the problems occurring. In the failures I have so far it is not due to syntax errors as the program I'm having problems with runs fine using other tools to get it on the SBC. I'm guessing it's some problem of characters overrunning either on the SBC or pigFX itself. I did double the UART buffer size, but that didn't help. Looking at the code it's not clear exactly how to read the SBC echos on a per character basis and maintain the UART buffer on pigFX. If you give me some hints I can see what I can do on my end. But I wanted to report what I've found so far.
3. Minor nit, but just so you know. I hacked in a config entry to allow me to set the default foreground color. However, I noticed that it does not affect the cursor which remains white - at least most of the time. I think it has to do with the way it reverses the background color. This might be a bit of a tricky fix looking at the code. But another case of letting you know. FYI I wanted to set the foreground to dark green to look a bit like a vintage green phosphor CRT. The white cursor sort of breaks the vibe :).

I'm using my 6502 SBC->level shifter->RPI Zero's GPIO UART. The board is running a version of EhBasic. Things are mostly working just fine.

Thanks again for your efforts.

[chregu82]

Please check out my recent changes.

1. The used FAT driver only supports 8.3 filenames. Where did you see, that it actally would support the full name? It's now possible to load the files with long names by using their 8.3 representation like they get listed with the dir command.
2. Transfers don't block the rest of the software now. So it would echo what gets sent. Please try.
3. The way the cursor works is quite simple. Every pixel at the cursor position gets bitwise inverted. So black becomes white, etc. There's no easy way to change this, especially if you always want to see what's on the cursor position.

[rkincaid]

My understanding of FAT32 is that it maintains BOTH a long and 8.3 filename in the directory for backwards compatibility. (https://www.ntfs.com/fat-filenames.htm) So long windows filenames will work on older software that don't support it. There is a comment I saw in fat32.c as follows: // Is it the volume label or a long filename entry (if so ignore) I tried to just comment this out the corresponding check but then it didn't recognize any USB devices, so it may be a catch 22 for this implementation. Either way, the reported 8.3 filename when you list the transfer directory doesn't seem to work when you try to transfer the file. Maybe it would work if I formatted as FAT16? But then I'm not sure my Mac will recognize it. I may try it anyway.

Thanks for the update on transfers

Your description of the cursor problem is what I thought was the case and why I thought it would be tricky. One could replace the XOR with restoring a screen save of the cursor location and paint the cursor with the foreground color and repaint the save as needed (maybe for non-blinking cursors only). Not as quick and as elegant as the XOR, but the code is doing something similar already for backspace and cursor movements I think.

[chregu82]

It's not a problem of fat32 or fat16. It's the used fat driver in PiGfx which doesn't care about the long names. It's not implemented. I had no problem transferring a long filename file with it's 8.3 name. Can you give me more info about that? What's the name of your file (long and 8.3)? What's its size?

[rkincaid]

Ok, a subtle "cockpit" error on my part :(. Turns out that Mac's "extended attributes" for filename.text can show up as "_filename.ext" - at least for 8.3 converted names. Such files are NOT the actual basic file. On the Mac I ran "xattr -c *" on the transfer directory and these files went away. The remaining 8.3 converted filenames are real and seem to transfer as expected. So you'll either need to make sure to document folks should be careful on MacOS and not save files in the transfer directory with extended attributes or maybe find a way to filter them. It might be as simple as ignore files beginning with underscore, but I'm not sure I'd depend on that. Or just warn folks to not try to transfer such files as they aren't what you think.

The new async transfer code seems to be working well. Thanks for that. I had managed to figure out a one-line patch for uart_send_ascii_file to accomplish the same thing, but your implementation is probably much better and more informed.

I'll bow out for a while and let you guys get back to whatever you have planned next :). But let me know if you have anything specific you'd like me to test.