Should subxt have a branch for each polkadot release to allow easy integration with other Substrate crates?

lexnv commented 2 years ago

If I understand correctly, there is a need for tracking which subxt versions support which Polkadot versions, such there is no confusion about their compatibilities.

This is indeed a nice feature to have, but I'm wondering if there's another way of doing it than keeping track with tags. If subxt x.0 supports Polkadot-0.9.x, since we strive to keep backward compatibility, the subxt x+1.0 should be compatible with Polkadot-0.9.x and Polkadot-0.9x+1.

One question would be if we wanted to add the polka-0.9.x and polka-0.9.x+1 tags to the latest release. Since subxt can also support dynamic queries, any future release should be compatible as long as the submitted extrinsics are accordingly modified.

Another option would be trying to automate tag publishing using the CI.

However, subxt could be used for some other chains in the future, leading to a matrix of tags being maintained.

@jsdw What do you think?

niklasad1 commented 2 years ago

The simplest would be match the polkadot release cycle and for every polkadot release have a branch of tag that ensured that works/compiles for polkadot-v<x.y.z>. Then we would need to change the substrate branch to polkadot-v0<x.y.z> for folks not have to patch substrate

If it could be automated via a GHA I'm down for it.

jsdw commented 2 years ago

Then we would need to change the substrate branch to polkadot-v0<x.y.z> for folks not have to patch substrate

I don't quite follow this bit? Right now we used released versions of substrate crates and of course these may eventually fall behind, at which point we need to update them to support that version of substrate and corresponding polkadot releases.

The ideal scenario I think would be that we maintain a compatibility table that, for each subxt release, has the lowest and highest versions of polkadot/substrate it's expected to work against.

The question is, how can we get there?

Finding the substrate commit that each version of polkadot was built off (going back eg a year), and running the subxt tests against those commits, would give us some confidence that subxt would work against those versions of Polkadot. We could probably write a tool to automate this process of (get polkadot release, find substrate commit, get/compile binary for that, run subxt tests against it)
Running the examples (which are based off polkadot but use fixed metadata) would help a little, though mainly I think we'd be looking for compile-errors-or-not here to check for compat.
There might always be some manual knowledge we need to apply too.

Perhaps a first step would be to write a tool/script to list the substrate commits for each polkadot version going back a while (maybe capped to a year), and then run the subxt tests against each of these. We could leave this sort of thing running on build-host to build up the list of supported versions for some version of subxt we're planning on releasing, and just indicate that support may go back further if it works as far back as we bother testing.

Finding the highest supported version would mean checking new polkadot releases against all subxt releases, and I'd suggest we don't do that for now as it seems overly arduous.

What I think we could end up with given this first step (which is fairly automated at least) would be something like:

Compatibility table

(Note: older versions of subxt likely support newer versions of polkadot/substrate than tested and displayed here. In addition, support for a given Polkadot release is inferred from the subxt tests passing against the corresponding substrate commit that it was built off, and may not be 100% accurate).

Subxt version	Polkadot versions supported	Substrate commit range supported
0.24	0.9.16 - 0.9.23	123abc - 456abc
0.23	0.9.15 - 0.9.21	bcd134 - acd076
...

niklasad1 commented 2 years ago

I don't quite follow this bit? Right now we used released versions of substrate crates and of course these may eventually fall behind, at which point we need to update them to support that version of substrate and corresponding polkadot releases.

It might fail to compile if someone for instance is targeting a substrate branch such as "polkadot-v0.9.12` then also is using some published version of subxt in the same library/application/crate i.e, with conflicting versions of the substrate crates used by both. In that case one have to patch all substrate dependencies in subxt and it may not compile.

So you are talking about compatibility with substrate node which is another story such a table would be great but it's something different :P

FWIW, I have seen that many parachains and such are using their "node" impl and subxt in the same repo and then they might run into issues when do so which I tried to explain :point_up:

So I guess @yjhmelody has to elaborate why having specific branches for the polkadot releases is useful :P

yjhmelody commented 2 years ago

Just regard subxt as a component of substrate. I focus on subxt's functionality not its compatibility.

In more simply words, Pushing same tags with substrate maybe is enough.

niklasad1 commented 2 years ago

Just regard subxt as a component of substrate. I focus on subxt's functionality not its compatibility.

In more simply words, Pushing same tags with substrate maybe is enough.

I don't understand what you mean by "functionality vs compatibility", can you elaborate please?

subxt doesn't "really rely" on the functionality in substrate I would claim to it just needs the correct metadata and then to construct extrinsics to be submitted via RPCs. Obviously, subxt might break when changes occurs such as some type in RPC APIs changes in substrate and that's why subxt checks that it's compatible with substrate master once per day. Thus, I believe @jsdw compatibility table would be sufficient then....

If your talking about runtime stuff in specific pallets that should be possible to fix via just to update the metadata.

yjhmelody commented 2 years ago

I'm talking about sp-* crates. They are version 4 dev. I need polkadot-0.9.xx. I use this crate in substrate node.

niklasad1 commented 2 years ago

substrate already has a dedicated branch for each polkadot release, such polkadot v0.9.28 but then I suppose you need patch all sp-*-crates in subxt for that work (which might not compile is something has changed)?

That makes sense to me

jsdw commented 2 years ago

I'm confused now; @yjhmelody could you describe what you're actually trying to do, and what the actual problem is that you're running into?

In order to publish subxt to crates.io, it has to rely on crates that are also published, so periodically we go through the (arduous) process of publishing substrate crates if we need to do so to keep subxt working properly.

Polkadot isn't published to crates.io and relies on tagged substrate versions straight from github. We cannot rely on those tagged versions in any published version of subxt. I suppose you're asking whether we can manually maintain a branch of subxt for each polkadot release that points directly to the substrate github repo with that polkadot tag?

The benefit as far as I can see of doing this would be simply to avoid some code duplication if you're using subxt alongside crates from substrate master.

The downside is that it would be locking you to using a specific subxt release for a corresponding polkadot one (and incolce some extra work on each polkadot release; possibly not much though; we'd have to try it to see I think).

yjhmelody commented 2 years ago

Polkadot isn't published to crates.io and relies on tagged substrate versions straight from github. We cannot rely on those tagged versions in any published version of subxt. I suppose you're asking whether we can manually maintain a branch of subxt for each polkadot release that points directly to the substrate github repo with that polkadot tag? The benefit as far as I can see of doing this would be simply to avoid some code duplication if you're using subxt alongside crates from substrate master.

Yeah, This is what I want to say.

jsdw commented 2 years ago

Gotcha!, thanks for comfirming!

If it's common for projects using subxt to want to integrate it with substrate crates (rather than write code which is somewhat more separate and uses subxt to help communicate with some other node) then this is definitely something we'd consider doing. Let's leave the issue open (and perhaps I'll tweak the question) to try and gather more thoughts on it!

The main limitation would be that I think we'd just, for a given polkadot release, crate a branch off the latest current subxt at that time. We wouldn't want to keep all such branches updated with each new polkadot release as that would be an exponentially growing amount of work.

yjhmelody commented 2 years ago

The main limitation would be that I think we'd just, for a given polkadot release, crate a branch off the latest current subxt at that time. We wouldn't want to keep all such branches updated with each new polkadot release as that would be an exponentially growing amount of work.

It's enough.

jsdw commented 2 years ago

Thanks for the info!

I don't think this is a super high priority thing for us at the mo; you can do the same thing yourself at the moment with a small patch on top of subxt to track whichever release branches you're interested in. We'd only want to step in if multiple people would benefit from us doing that work instead.

Also, we do have aspirations to start publishing Substrate crates too with polkadot releases. This will give us a much more solid path to take here; we can keep Subxt in line with substrate crate releases, which would be the best solution to this issue. Frankly though I'm not sure how far away that still is from reality though.

jsdw commented 2 years ago

https://github.com/paritytech/subxt/pull/688 has a similar need (though more around Subxt compiling ok against latest Substrate master).

drewstone commented 1 year ago

Can second needing this as well, currently subxt 0.25 does not work with any polkadot v0.9.3X branch. Is there some way around this?

error: failed to select a version for `parity-util-mem`.
    ... required by package `sp-runtime v7.0.0`
    ... which satisfies dependency `sp-runtime = "^7.0.0"` (locked to 7.0.0) of package `subxt v0.25.0 (https://github.com/paritytech/subxt.git#a80d6cfd)`
    ... which satisfies git dependency `subxt` (locked to 0.25.0) of package `webb v0.5.11 (/Users/drew/webb/webb-rs)`
    ... which satisfies path dependency `webb` (locked to 0.5.11) of package `eth2-pallet-init v0.1.0 (/Users/drew/webb/pallet-eth2-light-client/crates/eth2-pallet-init)`
    ... which satisfies path dependency `eth2-pallet-init` (locked to 0.1.0) of package `eth2_to_substrate_relay v0.1.0 (/Users/drew/webb/pallet-eth2-light-client/eth2substrate-block-relay-rs)`
versions that meet the requirements `^0.12.0` (locked to 0.12.0) are: 0.12.0

the package `parity-util-mem` links to the native library `parity-util-mem-ban-duplicates`, but it conflicts with a previous package which links to `parity-util-mem-ban-duplicates` as well:
package `parity-util-mem v0.11.0`
    ... which satisfies dependency `parity-util-mem = "^0.11.0"` of package `sp-core v6.0.0 (https://github.com/paritytech/substrate.git?branch=polkadot-v0.9.30#a3ed0119)`
    ... which satisfies git dependency `sp-core` of package `eth2-pallet-init v0.1.0 (/Users/drew/webb/pallet-eth2-light-client/crates/eth2-pallet-init)`
    ... which satisfies path dependency `eth2-pallet-init` (locked to 0.1.0) of package `eth2_to_substrate_relay v0.1.0 (/Users/drew/webb/pallet-eth2-light-client/eth2substrate-block-relay-rs)`
Only one package in the dependency graph may specify the same links value. This helps ensure that only one copy of a native library is linked in the final binary. Try to adjust your dependencies so that only one package uses the links ='parity-util-mem' value. For more information, see https://doc.rust-lang.org/cargo/reference/resolver.html#links.

failed to select a version for `parity-util-mem` which could resolve this conflict

niklasad1 commented 1 year ago

subxt 0.25 should work with polkadot v0.9.34 but if you have other dependencies that brings in an older version of parity-util-memory that won't work.

perhaps open another issue how to reproduce the issue you are facing but keep in mind that you need ensure that sp-core 7.0.0 and sp-runtiime 7.0.0 all of the place otherwise you will bump into this compile error.

Sorry for this parity-util-mem it's a PITA to deal with but now that https://github.com/paritytech/substrate/pull/12795 is merged, we should publish new versions of sp-core and sp-runtime to get rid of parity-util-mem in our dependency tree :)

clangenb commented 1 year ago

I would also be in favor of having a subxt branch referring to substrate git dependencies of the respective Polkadot release branch.

We use subxt for unit testing that XCM-transact calls that are constructed in our pallet are compatible with the current Kusama runtime. As subxt is only a dev-dependency, I have currently accepted to have duplicate substrate deps, but would naturally love to remove this duplication.

jsdw commented 1 year ago

Just to update on this; The plan going forwards as I understand it will be to release Substrate crates each time there is a Polkadot release (the exact source used in that polkadot release is what will be published). So, what we can do in Subxt then is to periodically do an update to keep in line with this, and record (either with a git tag or a version table in the README) which Subxt version corresponds with each Polkadot release (ie substrate crate publish).

clangenb commented 1 year ago

OK, I see so this is actually even more interesting. So the plan is that substrate devs, and even parachain teams that follow the release cycle closely, don't need to refer to git dependencies any more? Will Polkadot itself also feature a release using the published crates.io crates of substrate?

niklasad1 commented 1 year ago

Will Polkadot itself also feature a release using the published crates.io crates of substrate?

I don't think the polkadot binary itself will be published on crates.io but the crates in the polkadot repo will be published (there are some exceptions) but hopefully no more git dependencies :pray:

clangenb commented 1 year ago

Alright, thanks for the answers. Looking forward to this!

jsdw commented 1 year ago

Likewise; it will definitely make this stuff easier :)

paritytech / subxt

Should subxt have a branch for each polkadot release to allow easy integration with other Substrate crates? #626