build: :sparkles: automatic developer environment provisioning with gitpod - Githubissues

databricks / spark-xml

XML data source for Spark SQL and DataFrames

Apache License 2.0

500 stars 226 forks source link

build: :sparkles: automatic developer environment provisioning with gitpod #561

Closed edmondop closed 2 years ago

edmondop commented 2 years ago

This PR adds to spark-xml GitPod support and modify Github workflow actions so to run test on each branch.

srowen commented 2 years ago

This is non-trivial extra complexity. What does it do?

edmondop commented 2 years ago

Fair enough. I'll have to submit a PR for a bugfix, and as I tried to clone locally I immediately got into troubles because sbt 1.2.8 doesn't work with java 17. I then had to resort to SDK to manage multiple versions of java locally.

I therefore thought that automating developer environment setup could be valuable for future work and GitPod provides the possibility to create a pre-configured development environment with VSCode in the cloud, you can try to see what happens here:

https://gitpod.io/#https://github.com/edmondo1984/spark-xml/tree/gitpod

srowen commented 2 years ago

I see - I recall I had trouble updating SBT in this project, but maybe it's resolvable now. That would be a fine thing to try to fix directly, separately.

Spark itself only supports Java 11, though 99% works on Java 17 already in 3.2.0. When that's out, then it's reasonable to test vs later Java versions. For now just use a lower version, yeah.

I think it's kind of rare to want to test vs different versions of Java for the occasional contributors, as the build matrix covers supported versions already

edmondop commented 2 years ago

Thanks, I understand it. I changed the PR title, this is really just a way to get a pre-configured VSCode with the right jvm versions and the right dependencies available. Getting a dockerized developer environment will enhance the possibility of developers to contribute to open source repositories without paying the price of setting up the local environment, have you tried the link I gave you?

srowen commented 2 years ago

I don't use VSCode, and the build is really Maven here. I get it, but, generally I'd expect developers to use Java 8 or 11 to develop and test this library. I would like to fix SBT if that's all that stops this from working on 17, so it might not even matter

edmondop commented 2 years ago

Here is a recap of what I found:

I tried to update SBT to 1.5.5 but test would not pass anymore, I imagine due to some changes in the threading model. They would fail with an exception complaining that a Spark instance is already been created in the JVM
I can confirm that both with jdk 11 and 8 all the test pass locally (sdk install java 11.0.11.hs-adpt / sdk install java 8.0.292.hs-adpt)
I still think that adding the configuration to allow any external contributor to quickly create an environment in GitPod without going to the pain of installing sdk and multiple java versions it's a good idea and doesn't add extra complexity (but you are the repo owner, so I understand is your call)
On my first commit GitHub actions weren't run even if I had opened the pull request, that's why I had to change the triggers

srowen commented 2 years ago

I can trigger the GH actions. I think the issue is this is specific to VSCode (?) and just non-trivial complexity. I'd rather fix the SBT thing if that makes it more of a nonissue.

edmondop commented 2 years ago

No, the issue has to do with SBT and Java 17, which is the default version you get on a MacBook now. In order to check out and run from the terminal "sbt compile" and "sbt test", one needs to install sdk and manage multiple versions

srowen commented 2 years ago

Oh, hm, I may have a different version as I use brew. But doesn't it kinda use the version defined in the properties file of the build? I never really understood that ...