ericvergnaud
commented
2 months ago Development wise, it might be more effective to leverage ANTLR with Scala grammar https://github.com/antlr/grammars-v4/blob/master/scala/Scala.g4. This would avoid running the scala linter as a (slow) child process and/or recreating all the plumbing.
Is there an existing issue for this?
Problem statement
Some of the ETL code that has to be migrated to UC is written in Scala. We need to speed that migration up. There's https://scalameta.org/docs/trees/guide.html + https://scalacenter.github.io/scalafix/, that is relatively stable.
We can limit the scope to:
spark.read.XXX("fs://prefixpath/")df.write.save("fs://prefixpath/", ...)spark.table("...")spark.read.table("...")df.write.saveAsTable("...")spark.sql("...")with a call back to parent Python process for linting/rewritesdbutils.widgets.get(X)from a suppliedMap[String,String]dbutils.fs.callsProposed Solution
Implement linters/fixers for Scala
Additional Context
No response