additional Remote Interface Features

[mrdudz]

We mailed about this before, perhaps you forgot about it, so i am trying here :) This applies to both 1541U and U64 to some extend, both can benefit from some additional remote features, which would make it possible to run many more of the VICE tests automatically:

• can't mount .crt files over network (mounting itself works with the new command - but now you can no more remove the cartridge - we need a way to do this for the testbench)
• also i am not sure if mounting a cartridge, then running a prg file actually works right. it seems this will disable the cartridge? this worked at some point but behaviour is different now apparently. again, this is needed to run certain cartridge related tests.
• can't configure REU, GEORAM (we need commands to enable/disable each, and set the size)
• can't switch pal/ntsc
• can't mount a d64 and then send a prg file, it will always "run" the d64 apparently

(for reference: the mentioned mails had subject "U64 remote network interface", last one was on 25.7.2021)

[mrdudz]

Almost a year now since last lifesign... is there even still interest in fixing this? sigh

[GideonZ]

Hi! Yes, sure there is interest. As you might know, the U2+ needed to be redesigned. I can only spend my time once. I am very open to other people contributing. There is a design skeleton for the remote features. So far no sign of development from others.

[mrdudz]

Oh sure, its only that for the last two releases, i was already told "oh i am fixing this next for sure" and then a lot of random gimmicks were added once again - instead of fixing the one single important feature that is automatic testing. That pretty much implies lack of interest to me. But what do i know :) (Yes i realise adding random things is a lot more fun to do)

[GideonZ]

Hmm... I see your point. The things added were not entirely random; they follow a theme. Storage. Next theme is connectivity. But as said before; I understand your point. I'll start working on it as soon as the U2+L design has been signed off for mass production.

[daglem]

I attempted to run parts of the VICE testbench on a real C64 via a 1541U2. This fails since the debug register is seemingly only accessible on a U64 (code for SOCKET_CMD_DEBUG_REG inside #ifdef U64 in socket_dma.cc).

Is the debug register a U64 only thing, or can you simply move the code for SOCKET_CMD_DEBUG_REG out of the #ifdef U64?

BTW I see mention of a REST API at https://ultimate64.com/ProjectStatus - is this supposed to replace the socket API?

P.S. You may wonder why one would want to run the VICE testbench on real hardware. This could be very useful for testing of hardware replacement parts - personally I would use it to test https://github.com/daglem/reDIP-SID.

[GideonZ]

I attempted to run parts of the VICE testbench on a real C64 via a 1541U2. This fails since the debug register is seemingly only accessible on a U64 (code for SOCKET_CMD_DEBUG_REG inside #ifdef U64 in socket_dma.cc).

Is the debug register a U64 only thing, or can you simply move the code for SOCKET_CMD_DEBUG_REG out of the #ifdef U64?

It is a U64 only thing, because on a real C64 this location has a SID chip and whether the SID chip is actually mapped depends on the PIO registers $00/$01. These cannot be known from the cartridge port. So, reading will be impossible, writing could be implemented using a snooping mechanism, but is not guaranteed to work as writing to RAM location $d7ff will also trigger it. For this reason, the debug register is not implemented in hardware for the 1541U2.

BTW I see mention of a REST API at https://ultimate64.com/ProjectStatus -

is this supposed to replace the socket API?

Yes. Although, in the upcoming 3.11 both interfaces will still be there, as in a grace period.

P.S. You may wonder why one would want to run the VICE testbench on real hardware. This could be very useful for testing of hardware replacement parts - personally I would use it to test https://github.com/daglem/reDIP-SID.

Nice! Of course you could test these replacement parts in the U64.

Gideon

Message ID: @.***>

[daglem]

OK, that's unfortunate. Could you conceivably just read/write $d7ff directly, and rely on the fact that any value written to the SID chip will be living on the SID data bus for some time? I don't know what is supposed to happen on writes to $d7ff, but if bus snooping is required, could the value of the PIO registers also be tracked by snooping, in order to not trigger on writes to RAM at $d7ff?

Sure, the U64 could be used to test SID replacements, but not 6510, VIC-II, or CIA chips. The 1541U2(+) would be ideal for that, if it should somehow be possible.

In the mean time, perhaps I'll try to ask someone I know with a U64 to run SID tests - I already ordered a 1541U2+L in addition to the 1541U2 I already have, so I think I bought enough stuff from you for now :-)

[daglem]

Thinking some more about it, I guess the value of the debug register doesn't have to be read from $d7ff at all? If I understand correctly, it is only read via the socket API, and can thus be read from some internal location.

With that out of the way, would the "only" remaining challenge be to do the snooping for writes to (SID) $d7ff, as outlined above?

[GideonZ]

OK, that's unfortunate. Could you conceivably just read/write $d7ff directly, and rely on the fact that any value written to the SID chip will be living on the SID data bus for some time? I don't know what is supposed to happen on writes to $d7ff, but if bus snooping is required, could the value of the PIO registers also be tracked by snooping, in order to not trigger on writes to RAM at $d7ff?

Unfortunately, the value of the PIO registers does not appear on the bus and cannot be snooped. It would be by far easier to specify a different location for the debug register, in the I/O space itself, for instance. But, if only writing is required, then snooping $d7ff is probably enough, although tests that will write to RAM under I/O will fail. If these don't have your focus, it'll be fine. Note that esp. for SID stuff, it must be considered that writing to $d7ff is the same as writing to $d41f !

In the mean time, perhaps I'll try to ask someone I know with a U64 to run SID tests - I already ordered a 1541U2+L in addition to the 1541U2 I already have, so I think I bought enough stuff from you for now :-)

From a principle standpoint, I cannot agree with your latter statement. ;-) Thank you for ordering an Ultimate-II+L.

Message ID: @.***>

[daglem]

OK, so you'd simply need a "shadow" 6510 emulation in order to keep track of the I/O registers? ;-)

Jokes aside, if the only functionality of $d7ff is to store a value, it would IMO be very useful if you could implement snooping of $d7ff. I don't think that writes to $d7ff RAM would be any real issue for tests, and I don't see any problems with tests concluding by writing a value to the SID bus (unused register $1f) either.

Is there anything else going on with $d7ff except for storing a value? If not, can you please implement snooping and socket API read/write for the 1541U2? :-)

[mrdudz]

It would be by far easier to specify a different location for the debug register, in the I/O space itself, for instance. But, if only writing is required, then snooping $d7ff is probably enough, although tests that will write to RAM under I/O will fail.

$D7FF has been carefully choosen and none of those problems exist :) Snooping the bus for whatever is written to $d7ff and making that readable via the debug interface is all that is needed. Infact, it was my impression that this is exactly what U64 does - and i also thought 1541U2 does just the same.

As for moving to a REST API (whatever that is and why it is needed - seems weird and unneeded to me) - please fix all the long standing issues with the remote stuff first. Please.

[GideonZ]

But, if only writing is required, then snooping $d7ff is probably enough, although tests that will write to RAM under I/O will fail.

$D7FF has been carefully choosen and none of those problems exist :) Snooping the bus for whatever is written to $d7ff and making that readable via the debug interface is all that is needed.

It fails when a program writes to the RAM location $D7FF, because the debug register will also be triggered then.

Infact, it was my impression that this is exactly what U64 does - and i also thought 1541U2 does just the same.

No, the U64 has it mapped in the I/O space at $D7FF; it is part of the PLA decode.. So when I/O is mapped out, so is the debug register. The U2+ does not have the debug register at this point. That is what this discussion was about.

As for moving to a REST API (whatever that is and why it is needed - seems weird and unneeded to me) - please fix all the long standing issues with the remote stuff first. Please.

Long standing issues with the remote stuff? Could you elaborate? You mean missing features?

The remote control port 64 will be deprecated. Additional features will not be added to the remote control port, because it was never meant to be an official interface. It was never designed to return data, there is no versioning, no documentation... it just "exists".

The API is functional now, but not all calls have been implemented yet. This is work in progress. Going fast now, so I am expecting to have something usable within a month or so. This API will be fully documented and stable, and the features you need will certainly be part of it. Feel free to elaborate on the features you need (possibly again), so that I can check if this will really be the case.

To give you an idea: it was not possible to start a SID file through the remote API. This is now working fully. The same will be true for CRT files, TAP files, PRG files and so on. Configuration (settings) can also be set and retrieved. And of course you can create, mount disk images and the such, not limited to D64.

Message ID: @.***>

[daglem]

It fails when a program writes to the RAM location $D7FF, because the debug register will also be triggered then.

I think it is much preferable to have a debug register wihch is (also) modified on writes to RAM $d7ff, as opposed to not having a debug register at all. After all, if I understand correctly, the debug register is only storing a value which can later be read by the socket API (and I guess later the REST API).

In the unlikely event of a test program which writes to RAM $d7ff, well, that test program probably wouldn't work. But all the 99.9% other test programs would.

[GideonZ]

In the unlikely event of a test program which writes to RAM $d7ff, well, that test program probably wouldn't work. But all the 99.9% other test programs would.

Agreed

Message ID: @.***>

[mrdudz]

It fails when a program writes to the RAM location $D7FF, because the debug register will also be triggered then.

That is no problem. Tests dont write there by accident :) And we actually already proved the testbench is "clean" from such accidents - since it works on the C64RMK2, which implements the debug register exactly like this (any write to $d7ff goes to the debug register). With Chameleon its not even a register, but i use the Chameleon MMU to map some RAM there :)

No, the U64 has it mapped in the I/O space at $D7FF; it is part of the PLA decode.. So when I/O is mapped out, so is the debug register. The U2+ does not have the debug register at this point. That is what this discussion was about.

I was under the impression it was implemented by "bus snooping" the same way the 1541U can handle the SIDs. Infact, i think that is how you explained it to me years ago when it was added :)

In any case: implementing it like this is perfectly fine

Long standing issues with the remote stuff? Could you elaborate? You mean missing features?

Not only missing features, but also stability problems. The current interface crashes randomly and then U64 (or 1541U2) requires a powercycle. It's still not possible to run the testbench unattended for that reason.

[GideonZ]

Okay, I will do some repeat tests on both the API and the socket interface. As for the socket interface: Does this happen when you don't run any program and simply keep reading the debug register? Or does it only happen in combination with other stuff? And how long does it need to run before it happens? Does it run into a panic?

[mrdudz]

As for the socket interface: Does this happen when you don't run any program and simply keep reading the debug register? Or does it only happen in combination with other stuff? And how long does it need to run before it happens?

No idea really, i just notice i come back and everything hangs. The only way to reproduce it i can offer is: run the testbench, and it will happen eventually, anywhere in the next 6 hours :) Its really random unfortunately.

Does it run into a panic?

I wouldnt know how to see this. However it is obvious that at this point the "application" crashed, the network is dead, and the menu interface too.

[mrdudz]

Any progress here?

[GideonZ]

Yes, in fact there is quite a bit of progress.

• The http daemon is running
• The API structure has been defined
• Many commands have been implemented
• There is a UDP based discovery protocol that has been added
• The new ACID64 already supports the new API.

[mrdudz]

So where is the documentation, and the firmware update? :)

[GideonZ]

I think I didn't mention it was finished. I wrote that a large number of commands have been implemented, but certainly not all yet.

Documentation; there is a preliminary design document in a fork, but it wouldn't make sense to reference it at this point.

Firmware update is something you can build yourself, from the cli branch. I am not going to provide versions to the public that are incomplete or even untested.

[mrdudz]

ok, so no. too bad.

[daglem]

Good to hear that there is progress on the new API. Please don't forget about the debug register for the U2(+) :sweat_smile:

[markusC64]

I think I didn't mention it was finished. [...] Firmware update is something you can build yourself, from the cli branch.

Does not build for me:

Compiling mdio.c Compiling sid_coeff.c Compiling nios_main.c Compiling alt_malloc_lock.c Compiling alt_do_ctors.c make[1]: No rule to make target server.o', needed byoutput/ultimate.out'. Stop. make[1]: Leaving directory `/tmp/1541u2/target/software/nios2_u64' make: [u64_updater] Error 2

At least one file is missing in the git repository, that is "server.c" or "server.cc".

[GideonZ]

Did you do git submodule init and git submodule update ?

Message ID: @.***>

[markusC64]

Thanks for the hint. I have to check if theese are contained in the build enviroment. In my local repository I do have them.

[mrdudz]

Some months later.... i'll have to PING again :) We have a bunch of tests to fix/update/enhance in the VICE testbench, which would be really a lot less annoying to do if we could run them automatically on a real C64 (via U2). :)

[mrdudz]

@GideonZ another ping :) I'm just running the testbench for all emus and devices.... and Ultimate64 is the last one left - would be nice to gather some results from current firmware. So what is the status of this stuff?

[GideonZ]

About to release 3.11 alpha, which has Wifi and a start of the remote interface. The documentation of what is already there can be found here:

https://1541u-documentation.readthedocs.io/en/latest/api/api_calls.html

I am very well aware of the lack of the debug register readout. I haven't forgotten about it.

Message ID: @.***>

[mrdudz]

I am very well aware of the lack of the debug register readout. I haven't forgotten about it.

OK :) I will try another test run with the old remote stuff then

edit: done... its a bit tedious without being able to change options from remote :) There also seems to be some problem with mounting cartridges - it hangs a lot at that (so i had to filter out those tests). Generally network seems more stable though, it ran the other tests without hang.

Results: https://sourceforge.net/p/vice-emu/code/HEAD/tree/testprogs/testbench/results/u64.html

[GideonZ]

I added reading and writing to the debug register this morning. I will release a 3.11alpha shortly.

[GideonZ]

:) There also seems to be some problem with mounting cartridges - it hangs a lot at that (so i had to filter out those tests).

You did this with the API or with the old port 64 debug stuff?

[mrdudz]

You did this with the API or with the old port 64 debug stuff?

All with the old stuff of course :)

[daglem]

About to release 3.11 alpha, which has Wifi and a start of the remote interface. The documentation of what is already there can be found here: https://1541u-documentation.readthedocs.io/en/latest/api/api_calls.html I am very well aware of the lack of the debug register readout. I haven't forgotten about it.

Llooking forward to this! :+1:

Regarding the stability issues @mrdudz reported above, can this somehow be related to #364 ?

[GideonZ]

Possibly!

[xlar54]

saw the api. WOW. This will be great.

[mrdudz]

saw the api. WOW. This will be great.

To be honest - to me its just overcomplicated syntactic sugar that will make it (a lot) more effort to do what i need with it. Will be interesting to see how much my current code will be blown up by this :)

[GideonZ]

I agree that it is more complicated than a binary stream. However, "overcomplicated" is just one possible opinion. Adhering to a standard also gives great benefits. It would, for example, make it a lot easier to use JS to perform user actions. Whether your current code gets blown up, well, source code wise: just use one of the many libraries that exist to do http/url stuff. How big your executable is, is rather irrelevant nowadays. What do you think, @xlar54 ?

[mrdudz]

It would, for example, make it a lot easier to use JS to perform user actions.

Sure. It makes it easier to use in those hipster situations :) I don't see the use case for this myself :)

just use one of the many libraries that exist to do http/url stuff.

"just". sure. need to learn a library i don't use otherwise, need to deal with all kinda of issues related to this, and last not least need to package that library because its not standard C. It's just a lot of effort with (to me) zero benefits - a solution to a problem that never existed :) It will be interesting to see if ever something non trivial will be made that would actually require this sort of thing, and wouldn't work just as good with a simple package based interface.

Anyway, this is seriously off topic here, pointless discussion. It's not like this is going to change anyway :)

[GideonZ]

@mrdudz The player ACID64 has been tested with this API, using three different libraries (crates): attohttpc, hyper and rekwest. All three work. Given, these are Rust crates, not C libraries, but that's understandable as many people move from C to Rust nowadays for very good reasons.

Anyway, I totally understand that from your point of view this is overhead. But I think a lot of creative things could be created using this API, in combination with some html, css and JS.

[xlar54]

saw the api. WOW. This will be great.

To be honest - to me its just overcomplicated syntactic sugar that will make it (a lot) more effort to do what i need with it.

What is your use case? Not sure what your application is. Here's some others off the top, where I can see the benefits:

SID & MOD demo scene parties, remote playing - 24/7 SID radio baby! Remotely rebooting a BBS (as they often need rebooting) Remotely downloading code for testing All kinds of automated machine demos Automatically pull down and modify configs, and compare for issues (really helpful for supporting users) Powering off your machine remotely... another one for the sysops Vice (or any emulator) could mirror the U64 or vice versa by transfering data.. could aid in debugging Another big one - OTA updates. No reason these devices couldnt pull down the latest update on their own

The readmem and writemem has my mind buzzing..

Will be interesting to see how much my current code will be blown up by this :) If your work is open, Im sure folks would help you modify it to use the new architecture if adapting is your bigger concern. id be glad to help

[mrdudz]

None of this requires the new API, the old stuff worked just fine (if the network wouldnt crash and hang randomly like it did in the past - but thats an entire different problem). Actually i think all of this could be done with my years old tool :)

[GideonZ]

True - to some extent. For many functions it was however better to write the data to a temporary file (or two temporary files in case of the SID player), and then run the generic function "do this with this/these file(s)". The same function that is also called from the file browser. All arbitrary code for doing it either with a file OR with a raw buffer can (and will be) cleaned up in the future when this raw TCP support will cease to exist. Not having to maintain different variants of similar functions improves maintainability.

Another issue with the buffer approach is that these calls will cause memory fragmentation. This is a likely cause of crashes over time. When memory cannot be allocated anymore due to fragmentation, the application will crash. It's inevitable on an embedded platform without MMU. The HTTP calls also use some small amounts of memory; for the meta data only. The actual content is written to a file on the ram-disk, which is -by definition- able to handle fragmentation. Of course, it would have been possible to write files to the RAM disk from the raw TCP interface (or through FTP) and then send a command that references this file.

Then, not all functions were actually available through the TCP debug interface. I did see it as a hurdle to implement textual interaction, e.g. for setting / getting configuration parameters in tree-structures in a non-standard way. It's like re-inventing the wheel. It's logical to do this with JSON. The de facto standard. Would it have been possible to simply send JSON over the raw TCP interface; sure... Is it logical? No.

Then there ease of testing during development. It is totally possible to send commands from VS Code, using the ReST API plugin. Or, you can use curl from the command line, or you can use Postman, or... Whatever you choose, you don't have to write any tool yourself.

Another take on this is the context from which these operations can take place. From a PC, yes, it doesn't matter to use raw TCP once the tools have been developed. From a web-environment, however, it makes much more sense to use the HTTP protocol as it may not even be allowed to send data to a random port. This opens the way to wrap functions in a gui, or maybe do smart things in a browser, like inspect memory, or even have a run-time debugger / assembler in the browser. It opens the way to creativity, and that's what we want. Creativity! This is exactly what has @xlar54 's mind buzzing. 😊

[daglem]

Just don't forget about the debug register for the U2/U2+/U2+L :sweat_smile: I noticed that the documentation for GET /v1/machine:debugreg and PUT /v1/machine:debugreg says "This is currently an U64-only call."

[GideonZ]

Probably not possible for U2, due to size constraints... But for U2+ it should be possible.

[mrdudz]

Any news on this?

[GideonZ]

All the wishes have been implemented, except for the debug register in the cartridges.

[mrdudz]

Cool! Did someone make a cmdline tool to use those things yet? :)

Would be nice to have debug-register in 1541U(2+) soonish too - that would allow some really interesting things

[GideonZ]

Euh... curl?

[mrdudz]

Of course i mean something that hides all of the weird stuff and lets me work with sane parameters, without having to know the internal workings of the interface. Something like https://csdb.dk/release/?id=189723 or https://csdb.dk/release/?id=224409