search

tweag / sparkle

Haskell on Apache Spark.
BSD 3-Clause "New" or "Revised" License
447 stars 30 forks source link

Forward SparkleMain.main's "String[] args" to the Haskell main function #58

Closed alpmestan closed 7 years ago

alpmestan commented 8 years ago

Currently, the Java main function (in SparkleMain.java) doesn't forward the CLI argument given to the application (you can pass those by simply putting them after the path to the .jar you want to run, when invoking spark-submit) to bootstrap and therefore to the Haskell main. While this is by no means blocking or urgent, this should be quite straightforward to do and this issue will serve as a reminder.

robinbb commented 8 years ago

I have a suggested design for this issue, and I will implement it if you like it @alpmestan and @mboes. My suggestion is this: don't expect to give the Haskell RTS arguments via the arguments to main(). This is because main() is never invoked on the data nodes; it is invoked only on the master node, if I understand correctly from @alpmestan. Nonetheless, the Haskell RTS must be initialized on all nodes.

We currently do this via the static initializer for the Sparkle class. The Sparkle class is necessarily initialized because it's static apply method is used by Spark to invoke the Haskell functions. However, we can't pass arguments to the static initializer.

I propose that we use a Java property (sparkle.ghc-rts-string = "+RTS -N -RTS", for example) to initialize the Haskell RTS. The static initializer can look for this property, and then pass the string to hs_init(). The command line arguments to main() are not used. Thoughts? We could also do a more generic property, like 'sparkle.static-init-string' or something.

alpmestan commented 8 years ago

This issue wis more about simply being able to pass "normal" (as opposed to RTS) CLI arguments to a sparkle application, but having a way to pass RTS arguments as well would definitely be a good thing too! It indeed probably requires a separate solution though.

robinbb commented 8 years ago

Ah, I see. I was concerned with the GHC RTS arguments. I'll make a separate issue for that.

robinbb commented 8 years ago

I have a fix for this issue, but it is best to handle #63 first, as my fix is built upon that.

robinbb commented 8 years ago

Fixed in #69.