tweedegolf / sequential-storage

A crate for storing data in flash memory with minimal need for erasing pages
Apache License 2.0
99 stars 10 forks source link

Host side tooling? #20

Closed t-moe closed 5 months ago

t-moe commented 8 months ago

Hello Team Tweedgolf, I wanted to ask whether there is already some host-side tooling to read/write flash memory written by a target using sequential-storage.

I'm on esp32c6 here and I'm currently using espflash to interact with the target. But I could also imagine to build such tooling based on probe-rs, in order to target more architectures...

diondokter commented 8 months ago

Hey, never thought about that.

What would you want to see?

Like being able to create a file in some format and flash the data from that file to the device so you can give an initial config to the device?

Or more something to inspect the current values?

t-moe commented 8 months ago

My usecase: I want to configure my iot devices with a series of key-value pairs. Normally the config values are updated only over the air. But sometimes devices break in the field and are no longer reachable via internet. In that case the devices are shipped back to us and it would be nice if we could inspect their configuration with some host-side tools.

If I were on std, I would probably use something like https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-reference/storage/nvs_flash.html , which provides some python scripts on the host side.


Host side tooling wishlist, in descending priority :).

Since key/value types are pretty unrestricted and not always printable, I understand a CLI might be difficult to realize...

diondokter commented 8 months ago

Right, ok, then we're on the same wavelength. Cool!

You already mentioned the major problem in your last sentence. The code doesn't know anything about the key and the value.

So unless there's a way to describe the format to the CLI, the only thing it could realistically do is print out the raw bytes of the keys and values. And even if we could describe the format to the CLI, it would be out-of-band to the actual implementation on the device which means it could get out of sync on accident.

So a standard CLI could do all these things, but only when everything is only dealing with the raw bytes. That might be ok... Would that be ok for you? I guess that if we let the CLI work with only json values and only numbers as the key, that it could be done, but that'd be way too restrictive to make part of this crate.

Another way to go about this is to create a crate that impls the norflash traits around a probe-rs session. You could then use a probe-rs session as the flash on your host and just call the same apis as your device would. This would not be a CLI on it's own, but it would allow you to create your own pretty easily. This can also solve the out-of-sync problem. Because you could have your datatypes and serialization in a shared crate.

What do you think about that?

t-moe commented 8 months ago

Hmm....., Yes. Implementing the norflash traits seems like the most compatible/reasonable thing to do...

And then, it would probably make sense to create a self-describing, serde-compatible data format, based on top of sequential-storage. And implement the CLI for that.

But, I really only wanted to know, whether there are some thoughts/tools in this direction. I don't have the expectation that you take action immediately here ;).

Thank you! (We can probably close this issue.)

diondokter commented 8 months ago

I'll keep it open because I do find it interesting. But yeah, I might not pick it up right away (or ever). Who knows

diondokter commented 5 months ago

I've thought about it and I don't think I can make a useful tool out of this...