Potential Kafka Integration

[zedgell]

I am considering creating a Kafka listener similar to Redis and would like to get your alls input on what you think of this.

[itowlson]

That would be amazing! We don't have docs for this yet, but there are a couple of samples floating around (e.g. https://github.com/fermyon/spin-trigger-sqs), and we'd be very happy to provide support and guidance if that would be useful.

[zedgell]

@itowlson awesome, I will try to get a working example up and running soon! I will post the repo in this chat once I get it created.

[radu-matei]

Let us know if there is anything we can help in the meantime, @zedgell!

[zedgell]

@radu-matei and @itowlson I think I got a working example but I am unable to test due to the below error on Ubuntu 20.04.5 LTS. This was an issue also on the pluginify plugin when used locally. Have any of you hit this issue before? repo is here https://github.com/zedgell/spin-trigger-kafka

❯ cd guest
❯ spin build --up
Executing the build command for component zedgell: cargo build --target wasm32-wasi --release
    Finished release [optimized] target(s) in 0.35s
Successfully ran the build command for the Spin components.
You're using a pre-release version of Spin (1.2.0-pre0). This plugin might not be compatible (supported: >=0.10.0). Continuing anyway.
/home/zachary/.local/share/spin/plugins/trigger-kafka/trigger-kafka: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /home/zachary/.local/share/spin/plugins/trigger-kafka/trigger-kafka)
/home/zachary/.local/share/spin/plugins/trigger-kafka/trigger-kafka: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /home/zachary/.local/share/spin/plugins/trigger-kafka/trigger-kafka)
/home/zachary/.local/share/spin/plugins/trigger-kafka/trigger-kafka: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /home/zachary/.local/share/spin/plugins/trigger-kafka/trigger-kafka)

glibc version

ldd --version
ldd (Ubuntu GLIBC 2.31-0ubuntu9.9) 2.31

[itowlson]

Ah, the Ubuntu pluginify binaries are built on the latest version, so I bet they are picking up a libc that is not included with Ubuntu 20.04. Looking at your repo, your GitHub workflow also targets ubuntu-latest, so your downloaded release is hitting the same problem.

The most basic fix is to run the spin-trigger-kafka GH workflow on Ubuntu 20, but then the pluginify step will hit the same problem you saw locally. If it helps I can update the pluginify build to use Ubuntu 20 instead of latest and do a new release. Or you can bypass pluginify in your build workflow, and instead use tar and shasum to build the plugin and manifest. Let me know!

[itowlson]

I should add... if it is easy for you to set up an Ubuntu 22 based dev/test environment (e.g. it's all containerised), then that would solve all the problems. I know this is not usually convenient - I stayed on Ubuntu 18 for the longest time because I had it the way I wanted it - but just wanted to mention it as an option.

[zedgell]

I can do that. Thank you!

[zedgell]

@itowlson and @radu-matei there is a working example here - https://github.com/zedgell/spin-trigger-kafka. I am unable to support SSL security at this moment due to an OpenSSL error when cross building. I hope to have that fixed soon.

[zedgell]

Scratch the above the ability to use key and cert have been added now. Using cross-rs and the vendored feature in the OpenSSL crate.

[itowlson]

Nice! This is really exciting to see @zedgell!