Compatibility with vintage basic / Terminal abstraction / Module Packaging

[brickbots]

Hi Hi!

This project has been incredible. I've used it as a base for a RP2040 based portable computer that runs basic based on two hardware reference designs: https://github.com/bobricius/PICOmputer and https://www.adafruit.com/product/4818

Along the way I've made some changes and I submitted two PR's that I guessed you may want (Load/Save ASCII and tweaks to make the code micropython/circuitpython compatible while still working in cPython).

Some of the other changes I've made are more substantial and I'm wondering if you'd be interested in them. They may not really fit with what you are doing with this project, so I thought before opening up PR's I'd be wise to post here to get your thoughts.

If you are interested in one or more of these, let me know and I'll open up PR's for them so you can review them. Thanks again for this awesome project, without it I'd never would have been able to have my fun with portable basic computers!

Terminal Abstraction

To work with the LCD display and keyboard of the hardware devices above, I had to change prints/inputs to the hardware specific features. A more robust solve here would be to make PyBasic use the python curses library instead so that hardware specific versions of this library could be used without modifying the base PyBasic code.

Module Packaging

Restructure the code into a PyBasic module which can be cleanly imported into other code bases to enable it's inclusion as a git submodule. This is pretty minor restructuring, just adding a directory to the repo, moving all the .py files there, adding and init.py and changing a couple imports in each file. This would contain everything in a single PyBasic module/namespace. i.e from pybasic import interpreter interpreter.main() or import pybasic pybasic.interpreter.main()

Compatibility with vintage-y basics

Many computers from the late 70's and early 80's had a similar basic dialect based on MS Basic. There is quite a bit of software available in this dialect and it's very close to PyBasic. See: http://www.vintage-basic.net/index.html I've made changes to PyBasic to make it more compatible with vintage basic and can get a PR going if you'd like:

• Print element separator changed to ;
• Print will not print cr/lf if final token is ;
• Input prompt / variable separator changed to ;
• RND required argument. Negative values reseed with that value, Zero returns last value (not implemented), any positive number returns random float between 0 and 1
• DIM allows multiple array definitions seperated by commas
• DIM over dim by 1 to handle dialects that are 1 based and expect there to be an element = length

These allow the loading of many older basic programs without modification. I'm also working at multi-statement lines and a couple other bits with the goal of loading most of the software linked above without modification.

NOTE: This would be a breaking change with existing PyBasic programs...

[richpl]

Thanks for the input. I'm glad the repo has been useful for your projects. I'll take a look at the PRs over the next few days.

Regarding the more substantial changes, they all look pretty sensible to me. I haven't actively worked on the codebase for nearly three years, so it's hard to find the necessary time to get back into it again and make significant modifications. But the philosophy I've adopted is to try the make the code as generic as possible, so that folks like yourself can tweak it and use it in your own projects.

[JiFish]

Along with this I'd like to abstract all the print/input stuff in some way. I'm thinking some simple class that represents a terminal and gets passed into program when execute is called. Thoughts?

100% agreed. My intention was to use the below system replace any command that interacts with the terminal, but this is a much better way. I believe https://github.com/theinternetftw/xyppy uses this approach, so you might want to grab some ideas from there.

I'm super curious what you are thinking of for adding commands externally? ... And I think it might be able to be replaced by a dictionary of token categories mapped to functions.

This was exactly direction I was going, though I hadn't thought it out much further than that. I think there are some interesting design decisions to be made here. Since BASIC isn't a functional language, do commands and functions need to be dealt with separately? What about operators?

Compatibility with vintage-y basics

+1 from me. Anything that increases the available useful documentation for the language is a good idea in my opinion. I already PR some changes that increase compatibility, but I didn't go as far as breaking old programs. If @richpl is happy with that then I'm game.

[brickbots]

Hey Hey!

It seems like we might have plan coming together. I'm so excited to have some folks also interest in some of this same sort of functionality. Here's how I think we might be able to stage this, and I'd love to hear your thoughts on this @richpl as the master of PyBasic:

In Progress: @RetiredWizard has started some PR's for vintage basic compatibility. He's done most of the heavy lifting here and has a PR for file IO open. He's also got a nice implementation of DATA with ability to RESTORE. I think this is also a great candidate for a PR. This will solve two of the major feature deficits for compatibility, then it's just smaller keyword/function/syntax changes. I have a branch ready with these and once the File IO / DATA PR's are in, I can rebase to this branch and submit my PR. I think that will allow PyBasic to load and run many (most?) vintage type examples online.

Next Up: While getting the vintage compatibility worked out, I'd like to get a PR going for the modularization and terminal abstraction. I have a lot of this work already in my non-forked port, I just need to clean it up, document it and test it in prep for a PR. The intent of this PR wouldn't necessarily be to merge (as I suspect it will end up conflicting with the master branch that's taking on the Vintage Basic changes), but more to get a discussion going and get some feedback about the changes.
If we can reach some consensus about this, I can rebase the changes on the master branch and submit a mergeable PR (if the original one is defunct).

After that, or perhaps in parallel (@JiFish not sure if you want to take a run at this?): Restructure the basicparser / basictoken files to allow easier additions of non-standard basic commands (sounds, graphics, input) to support specific hardware cases. After looking at the code, I'm thinking this might not be too bad:

• Subclassing basictoken might be enough to add new keywords/functions
• Restructuring basicparser to use a keyword/function map should be doable, but it's probably going to break stuff along the way, so solid testing would be nice.
• Expanding regression.bas would be in order and maybe some sort of testing harness via the terminal abstraction so the input/output of regression can be automatically validated.

Longer Term: Memory optimization... I have some ideas for making the in-memory representation of a program much, much more compact. Compressing the tokens into a single byte array (there are fewer than 255 of them, even if the number almost triples), and putting all the non-token stuff (strings, data, name, integers, floats) into another zero delimited byte array. This is really only useful for memory constrained systems like micro/circuitpython, so it might not make sense to push PyBasic that far, as it's a useful learning platform due to it's really nice code design/readability. This would start to get into tricky areas and definitely make the whole code base a bit harder to follow.

At the same time I think we can solve the compound statement loop issue and allow renumbering of programs and/or numberless programs with just labels if desired.

[brickbots]

I've pushed PR #34 with some proposed ways to modularize an abstract the screen/keyboard stuff. Happy to hear any feedback either of these 👍

No rush on this as it's probably going to need refactoring once any other PR's are merged. In the meantime I'm going to play around with the testing harness and a curses version of the terminal emulator with more robust screen/cursor control and input and may update this PR as those come through.

[richpl]

All, thanks for the huge effort you are making. Personally I'm happy to break old PyBasic programs in order to increase compatibility with vintage BASICs (I based the interpreter on TI BASIC because that is the home computer I had back in 1982, but it is a very limited dialect).

The main thing I want to preserve is the relative simplicity of the interpreter, because that makes it easy for people to use in their own projects. I also want to preserve hardware independence so that the maximum number of people can use the interpreter on their own systems, they can tweak it for a specific platform if they wish in their own fork.

I'm relying on everyone properly testing their code before submitting the PR, I don't really have the time to regression test everything and the PRs are coming thick and fast!

[brickbots]

Thank you for being so open to these changes. I'm looking forward to seeing Adventure run on PyBasic first hand, although I've already gotten a peek from @RetiredWizard 's youtube videos!

I think PyBasic is a pretty darn good teaching tool, with lots of good examples of high-level CS stuff, so I'm keen to keep that clarity and conceptual neatness too. Hopefully we can keep the class interfaces/responsibilities nice and clear while allowing for cleaner re-use and even some exotic hardware (Artemis??)

[richpl]

@brickbots, @RetiredWizard, aware that I haven't addressed this issue yet.

In the meantime, now that your awesome work has increased compatibility with vintage BASICs, I've been able to port ELIZA and get it running. The PyBASIC version is in the examples folder.

I've also added some acknowledgements in the README introduction as I feel totally indebted to your efforts over the last few weeks.

[brickbots]

Hey Hey @richpl! Awesome to see ELIZA alive and well under PyBasic :-) Thanks for the acknowledgment, but you put out a great base to build on and it's been heaps of fun.

This issue may have run its course as I think most of the value has been implemented! Vintage basic compatibility is all merged in and seems to be working to ease porting. I've got that PR open to show some implementation ideas for the modularization and screen/keyboard abstraction.

Not sure if @JiFish is still interested in exploring some way to allow extended the language for specific use cases outside the main repo. I'll be doing some of this to add sound and some graphics for my handheld basic computer, so I'll be interested in seeing how it can be done while still allowing graceful updates of the PyBasic core. Maybe I'll open a specific issue for this to discuss if anyone else is interested?

[RetiredWizard]

Nice work on ELIZA @richpl! My version of PyBASIC for PyDOS is actually very close to the primary version now. I've held off putting in LEFT$ and RIGHT$ as they can be accomplished with MID$ and I'd like to keep PyBasic for Micropython as small as possible. I did find a parsing bug in my version through eliza, and the micropython stack size causes issues for complex statements so I had to break one line into two less complex lines (line 340), but I was able to get eliza running on the RP2040 micro controller very quickly!

I'm going to do some optimizations to speed it up and then post it to the PyDOS repository :D

[RetiredWizard]

One other thing I noticed, line 1 is a gosub 1300 which appears to be a subroutine with nothing but DATA statements. DATA statements don't have to be executed, they get loaded into the BASICData class data when the program is loaded so I don't believe line 1 is necessary.

[richpl]

Yes, I inserted the GOSUB because my original implementation required the DATA statements to be executed. I guess you must have fixed that?

I guessed that the GOSUB wasn't needed because I'd added a jump to the wrong line anyway! I've uploaded a fixed version to the repo

[RetiredWizard]

I was just looking at the commit history and noticed on the Eliza and Collaborator commit a few of the readme changes that went in with earlier commits were backed out. I'm thinking that was unintentional?

[richpl]

Damn, you're right @RetiredWizard, my bad. I'll see if I can recover them.

[richpl]

I think I've restored the accidental back outs. Grateful if you could quickly check. Sorry, I thought I was being careful with versioning.

[RetiredWizard]

Looks good to me