lfse-slafleur
commented
3 days ago @WouterSpaak @MichielTukker @MarkTNO @cwang39403 Curious to your thoughts! Feel free to invite anyone else in the discussion of course.
Open lfse-slafleur opened 3 days ago
lfse-slafleur
commented
3 days ago @WouterSpaak @MichielTukker @MarkTNO @cwang39403 Curious to your thoughts! Feel free to invite anyone else in the discussion of course.
Hi everyone!
Issue description
Currently we are in the process of converting a number of SDK functionalities that are already in https://github.com/Project-OMOTES/omotes-sdk-python to https://github.com/Project-OMOTES/typescript-sdk . This is proving cumbersome. Every release, there are a number of items (varying in size) that need to be implemented in both languages. This doesn't scale for multiple reasons.
Example of the items moving from mvp.3 to mvp.4:
Currently we support Python & Typescript, but if we would throw Java, Ruby or other (web/backend) languages in the mix this is going to progress from cumbersome to impossible to manage. Especially because some of the details are so finnicky.
Solution directions
Now I looked into bindings between Python & Typescript but they do not exist on the browser-side. This, of course, makes sense because Python is a C-based language while Typescript is Javascript/WASM based. On the backend, assuming nodejs is running, there does exist a good option:
Other attempts, also from Wouter, such as https://www.asyncapi.com/en, https://faststream.airt.ai/latest/ or https://typespec.io/ do not seem to completely fit. Either because the support (looking at you AsyncAPI) is lacking or because we need more than the technology provides us.
Alternatives have proven difficult because we need to do more than just communicate with a single API. The SDK also contains logic. Although we try to keep this logic simple and small, it has been growing steadily and we are at the point where having even 2 SDK's in different languages is becoming unruly.
Nevertheless, we do appear to have another option: Rust! There are bindings for rust -> {typescript, python} and many other languages:
Options
So currently there appear to be at least 3 options:
Now the positive I see for choosing Rust (option 3) over Python (option 2) is to not have to mess around with the Python interpreter. If we would do option 2, any application which uses an OMOTES SDK will also have to install Python, any of the dependencies we also require and make it available in the right way. Rust (Option 3) leads to a compiled binary which is very portable and contains all dependencies already in one big fat binary.
Side-note: Currently the python sdk also contains internal classes & code used by the orchestrator & workers: https://github.com/Project-OMOTES/omotes-sdk-python/tree/main/src/omotes_sdk/internal . I would propose to NOT port over this code as the orchestrator & workers will remain in Python of course. I am mainly referring to any common code and SDK/top-level code https://github.com/Project-OMOTES/omotes-sdk-python/tree/main/src/omotes_sdk and https://github.com/Project-OMOTES/omotes-sdk-python/tree/main/src/omotes_sdk/internal/common
Conclusion
Currently my preference would be to put in the investment of a couple of days to port over the Python SDK to Rust (option 3) and bind to Python and NodeJS/Typescript. This would follow the DRY principle and centralize all SDK logic so we only need to get the details right once instead of in many different languages. Also, we would not have to mess with the Python interpreter and dependencies which is why I prefer it over option 2. The downside is that we would have to invest in learning Rust for any of the SDK developers.
Timeline
After v0.5 and we would start with a small PoC to check if the bindings work as expected.
Do you see any other options or prefer one option over the others?