search

gateway-experiments / remote_kernel_provider

Base support for remote kernel providers
Other
3 stars 3 forks source link

[DISCUSS] Monorepo? #12

Closed echarles closed 5 years ago

echarles commented 5 years ago

I may be easier to have a single repo with all the source code for now (To be discussed). This would ease contributions.

kevin-bates commented 5 years ago

Thank you for opening this discussion! I've been struggling with this one in particular. We want to have separately installable providers, so the single repo would need to support multiple deliverables. This alleviates the "dependency payload" since its unnecessary to install the k8s or docker apis if you're only using YARN, etc.

We also talked (internally) about how releases would work, but I suspect its "okay" for a "Remote Kernel Providers" (note plurality) to use a single release number across all deliverables even though some releases may not affect every deliverable. In the end, I think I'd prefer a single repo.

Other things we want to "deliver" are kernelspec installation tools (for each provider). This would alleviate the OOTB kernelspec hassles (although may create issues of its own). I could definitely see a "util" package in remote_kernel__provider here that the other providers use to manage things like files, etc.

If your packaging "voodoo" knows how this can be accomplished, I think it would be a great way to go. We'd still want folks to be able to "house" their own "remote kernel provider" wherever they want, but I don't really see issues with that - just fyi.

echarles commented 5 years ago

Sure, these are classical discussion and consideration for release:

  1. I am not used to that, but pretty sure a monorepo can produce and push multiple artifacts to pypi, so it keeps the separation of providers correctly.
  2. It is indeed to me OK to release even if one provider is not changed. I prefer not having to maintain something like lib x.y is comptatible with lib w.z... and so on. You want version 0.3? so use 0.3 for all libs.

Not much voodoo magic to share I am afraid... more a matter of time...

Extending to the kernelspecs, we could even have a metadata like in package.json stating the minimal/maximum version number it supports.... Or we could have yarn_kernel_0.3.json .... but I prefer the metadata stuff. And we could also have a structure with version number subfolders...

The more control you bring on version number deps, the easisest the future Kernel install tools will be.

But as very first step, I am for the monorepo merge... but I am afraid it is all in your hands to take that road and implement it... :)

kevin-bates commented 5 years ago

We've now built two kernel providers (YarnKernelProvider and KubernetesKernelProvider) that derive from RemoteKernelProvider and reside in separate repos. The primary issue is development but I suspect the same issue would exist if these were all in a monolithic repo, since the structure, relative to import statements, etc, would still be that relative imports would not work outside of development. As a result, I'm thinking that we leave the repos separate at this point. This implies that RemoteKernelProvider will likely require frequent releases, but I believe the division of labor (including the kernelspec tool and resources) is setup such that that can be minimized.

kevin-bates commented 5 years ago

Sounds like things are going to stay as separate repos were remote_kernel_provider is essentially an abstract base provider. :smile: As a result, I'm going to close this issue.