Publish on crates.io

[brayniac]

Tracking ticket for publishing on crates.io:

src/common - pelikan-common - "Common internal libraries for Pelikan"
src/config - pelikan-config - "Component configuration for Pelikan"
src/core/admin - pelikan-core-admin - "Admin components for Pelikan"
src/core/proxy - pelikan-core-proxy - "Proxy components for Pelikan"
src/core/server - pelikan-core-server - "Server components for Pelikan"
src/core/waker - pelikan-core-waker - "Cross-thread wakers for Pelikan"
src/entrystore - pelikan-entrystore - "Maps protocol commands to Pelikan storage operations"
src/logger - pelikan-logger - "Logging framework for Pelikan"
src/net - pelikan-net - "Networking components and abstractions for Pelikan"
src/protocol/admin - pelikan-protocol-admin - "Admin protocol for Pelikan"
src/protocol/common - pelikan-protocol-common - "Shared protocol traits for Pelikan"
src/protocol/http - pelikan-protocol-http - "HTTP protocol for Pelikan"
src/protocol/memcache - pelikan-protocol-memcache - "Memcache protocol for Pelikan"
src/protocol/ping - pelikan-protocol-ping - "Ping protocol for Pelikan"
src/protocol/resp - pelikan-protocol-resp - "Redis serialization protocol for Pelikan"
src/protocol/thrift - pelikan-protocol-thrift - "Thrift binary protocol for Pelikan"
src/proxy/momento - pelikan-momento-proxy - "Pelikan proxy which allows Memcache and Redis clients to communicate with Momento" - build target: momento_proxy
src/proxy/ping - pelikan-pingproxy (?) - "Pelikan proxy which uses the ping protocol" - build target: pelikan_pingproxy
src/proxy/thrift - pelikan-thriftproxy - "Pelikan proxy which uses the Thrift binary protocol" - build target: pelikan_thriftproxy
src/queues - pelikan-queues - "Queues for cross-thread communication in Pelikan"
src/server/pingserver - pelikan-pingserver - "A simple ASCII ping/pong server" - build target: pelikan_segcache
src/server/segcache - pelikan-segcache - "A Memcache compatible server which uses segment-structured storage" - build target: pelikan_segcache
src/session - pelikan-session - "An abstraction on-top of streams for buffered bi-directional communication"
src/storage/bloom - pelikan-storage-bloom - "Pelikan bloom filter storage"
src/storage/datapool - pelikan-storage-datapool - "An abstraction which provides byte-addressable storage"
src/storage/seg - pelikan-storage-seg - "Pelikan segment-structured storage"
src/storage/types - pelikan-storage-types - "Typed storage for use in Pelikan"

Open questions/observations:

• do we want to publish the thrift proxy? (thinking no)
• should src/queues move to rustcommon? (thinking yes)
• should we use pelikan_ping_proxy or pelikan_pingproxy for "generic" proxy naming patterns?
• I believe we do want to use pelikan_segcache for the binary name (note: dropping "_rs" suffix) and allow install via cargo install pelikan-segcache
• should src/net or src/session move out to rustcommon? They feel like they could be useful in projects like rpc-perf, though we can depend on them through the pelikan- crates.
• Momento proxy may

And some hazards to consider:

• crates cannot be deleted, if we want to rename we will pollute our "namespace"
• namespaces do not exist, anybody could write and publish a pelikan- crate
• while the crate name (used for cargo install) does not need to match the name of the binary(s) that it provides, we should strive to make it easy for users

[brayniac]

Some additional notes on this: src/core/waker - moved to rustcommon, published as awaken

[swlynch99]

Some additional questions/notes:

• Do we want to version all of the related pelikan-* crates in lockstep? If not, what is the versioning policy?
• We should probably reserve the pelikan crate too - maybe as a virtual "install all the binaries" crate
• Do we intend for the various storages to be useful as standalone libraries? (e.g. for segcache) If so, we'll probably want to do a second pass over them so they are useful.
• With this many crates being managed in the repo we will probably want some automation to handle the publishing work. There are tools for this and we should look into using them.
• Owner on crates.io - we should probably make a crates.io user for the pelikan foundation, that way we can either have publishing go through github actions or have a team with publishing permissions.

[brayniac]

@swlynch99 - great questions

• I think we should move from the unified version for the workspace back to version per crate. It's going to make the version number meaningful for the individual binaries. We'll do semver for everything, current plan would be to release each as 0.3.2 (next up from current 0.3.1). https://semver.org/#semantic-versioning-specification-semver
• great idea, i'll grab that now with a 0.0.0 namesquatting crate
• yes, individual storage libraries should be useful in other projects. eg 'seg' storage should be usable within another application or library. I believe the 'seg' crate is already in good shape for this, but we'll take another pass across everything and improve docs/ergonomics
• if you have suggestions, please share. I personally didn't love cargo-release when I last used it to contribute to an org repo. If we don't know yet, I'll try to see what other large projects are doing.
• yes, we'll also want to figure a plan for RPMs, DEBs, etc where appropriate

[punkeel]

This work would make it easier to embed Pelikan (and especially segcache) in other services! 👍 Thank you so much for considering that!

When you get to this work, could you please consider reviewing the dependency chains in Pelikan? Namely, today: seg depends on (pelikan-)common which depends on boring (--> boring-sys) and (pelikan-)net. These introduce indirect dependencies that shouldn't be required to use segcache.

89 tackled this last point ^

[brayniac]

segcache storage crate has been published