Re-structure into combined lib & bin project

[sandreae]

When building an application with p2panda an emerging pattern is to deploy required schema to a local node on first start up. This can be done manually with the existing fishy cli, but it would be handy if developers could import the required functionality from fishy lib and do this initial deployment programmatically.

This PR explores splitting fishy into a lib module where core functionality lives, and a bin module where the cli application lives (and imports methods from the lib).

Thoughts and comments welcome! Idea was first raised in this issue: #3

Example use [deploy]

use std::path::PathBuf;

use fishy::lock_file::LockFile;
use fishy::Client;

// Parse lock file and get commits
let lock_path = PathBuf::from_str("schema.lock")?;
let lock_file = LockFile::from_path(&lock_path))?;
let commits = lock_file.commits();

// Create a fishy client (wrapper around a GraphQL client)
let client = Client::new("http://localhost:2020/graphql");

// Publish commits on external node
for commit in commits {
    let did_publish = client.publish(commit).await?;
}

Initial thoughts

• it's probably a bit contentious to expose a Client in this way as this is not exactly a core fishy concern, and likely something any app will be constructing elsewhere anyway. Open to other ideas here :+1:
• the reason for not simply exposing high level methods like deploy is that (like in fishy cli) one might want to expose user feedback along the way, as schema are deployed. For this you need closer access to the publishing process.

[Corban-Dallas]

Example with LockFile looks good for me. But it would be even better if we have an option for embedding somehow the content of schema.lock into build during compilation.

There are some valid points to consider:

• The schema used will be integrated into the application library, ensuring consistency of application and schema.
• The application rust library can be easily distributed and updated across different platforms without the need for synchronously providing or updating an external schema.lock file.
• This approach is more secure, as we can be certain that the schema.lock file will not be replaced by a malicious actor.

I didn't find a good practice of embedding file into compiled rust lib, but maybe it is possible with macros. Something like:

let lock_file = lock_file!("./schema.lock")?;
let commits = lock_file.commits.unwrap_or(Vec::new());

it's probably a bit contentious to expose a Client in this way as this is not exactly a core fishy concern, and likely something any app will be constructing elsewhere anyway. Open to other ideas here 👍

Indeed, the issue with the client is not simple. Creating yet another client from fishy lib to deploy the schema seems inappropriate. If developer is working with the node programmatically, one already have a client somewhere. Frankly, I don't see a good solution here. After some thought, I believe the existence of such a problem may tell us that the initial separation into library and cli is probably not the right approach.

If we decompose the task, all that fishy-lib has to do is to read schema.lock and convert it into [Commit], then fishy-cli do the rest. Not sure it is enough for independent cargo lib and also feels like something that can be used not only for schema deployment. For example - deploy commits from test.lock for node preconfiguration (with schema and published documents) for tests.

This functionality (read commits from *.lock) can be placed under p2panda::utils maybe. Then fishy-cli can just use it to read commits and deploy them by its own GraphQL client. It is not looks like best solution but as a possible compromise. This also will eliminate the need to add additional dependency from fishy to project.

I think this raises a good question about the availability of a convenient interface for interacting with the Panda node. Perhaps it is worth considering a wrapper around GraphQL with a convenient p2panda-related API? Although the GraphQL protocol is common, the developer must adhere to strict rules for working with the p2panda protocol, and having a client that takes on some of the work (including reading and executing commits from a file for schema deploy) can be a great help.

For now we have a missing part:

• Foundation: p2panda-rs
• Server side: aquadoggo
• Client side: ? some external GraphQL client

[sandreae]

Thanks for your thoughts @Corban-Dallas, really insightful stuff :+1:

Example with LockFile looks good for me. But it would be even better if we have an option for embedding somehow the content of schema.lock into build during compilation.

There are some valid points to consider:

The schema used will be integrated into the application library, ensuring consistency of application and schema. The application rust library can be easily distributed and updated across different platforms without the need for synchronously providing or updating an external schema.lock file. This approach is more secure, as we can be certain that the schema.lock file will not be replaced by a malicious actor.

Yeah, this is a pattern I've been wanting/exploring too. Think the benefits you outline are absolutely desireable.

Without adding any functionality to fishy it's possible like this:

let data = include_str!("schema.lock");
let lock_file: LockFile = toml::from_str(&data).expect("error parsing schema.lock file");

If we decompose the task, all that fishy-lib has to do is to read schema.lock and convert it into [Commit], then fishy-cli do the rest. Not sure it is enough for independent cargo lib and also feels like something that can be used not only for schema deployment.

Really agree with this, the most useful struct for developers to get their hands on from fishy is LockFile. As this implements serde traits it is easy to construct (as above) any way one would like. Then as you say, any client can be used to publish the contained commits (entries) to a node. We can consider if this should be exposed from p2panda-rs, I'm not totally sure about that as currently it's very much only focused on the core protocol.

I think this raises a good question about the availability of a convenient interface for interacting with the Panda node. Perhaps it is worth considering a wrapper around GraphQL with a convenient p2panda-related API? Although the GraphQL protocol is common, the developer must adhere to strict rules for working with the p2panda protocol, and having a client that takes on some of the work (including reading and executing commits from a file for schema deploy) can be a great help.

Yes, this is absolutely what we want. It's only a matter of capacity and higher priorities right now. We plan to have a rust crate for this comparable to https://github.com/p2panda/shirokuma. Eventually an abstraction over the query interface as well.

[Corban-Dallas]

let data = include_str!("schema.lock");
let lock_file: LockFile = toml::from_str(&data).expect("error parsing schema.lock file");

Didn't know about this way, thanks!

I'm not totally sure about that as currently it's very much only focused on the core protocol.

Yeah, sounds reasonable. Even if I proposed this, I am also dissatisfied with this solution. There is simply no suitable place where you can place LockFile for reusing for now.

Yes, this is absolutely what we want. It's only a matter of capacity and higher priorities right now. We plan to have a rust crate for this comparable to https://github.com/p2panda/shirokuma. Eventually an abstraction over the query interface as well.

Sounds cool! If there will be some p2panda client, the publishing entities from lock file can be part of it in the future. May be under feature.

[sandreae]

Seems we talked ourselves out of any changes in this direction :sweat_smile:

Would be interested to hear from @adzialocha though, before possibly closing this PR and putting it down to good research.

[adzialocha]

Would be interested to hear from @adzialocha though, before possibly closing this PR and putting it down to good research.

fishy is fairly powerful (automatically resolving dependency trees, handling key pairs, talking to node via GraphQL etc.) and as you already discussed, maybe that's all not what we actually need even though I also thought we should definitely start with fishy as a lib initially.

The thing where I would like to see this currently the most is actually in aquadoggo itself. Maybe something like this?

use aquadoggo::{Configuration, Node};
use p2panda_rs::identity::KeyPair;

let config = Configuration::default();
let key_pair = KeyPair::new();
let node = Node::start(key_pair, config).await;

let data = include_str!("schema.lock"); // super cool, didn't know about that either!
let lock_file: LockFile = toml::from_str(&data).expect("error parsing schema.lock file");
node.migrate(&lock_file).await;

We can publish all the data with our internal methods we have inside of our node, no need to go via the GraphQL API here.

Later it might be interesting to also add this to a client SDK, but somehow I see migrations currently rather a node concern (it is the starting point).

[adzialocha]

Throwing in some more ideas for aquadoggo here: https://github.com/p2panda/aquadoggo/issues/597

[sandreae]

These changes have been taken on by @adzialocha over here: https://github.com/p2panda/aquadoggo/pull/598

Closing this PR.