JaneliaSciComp / BigStitcher-Spark

Running compute-intense parts of BigStitcher distributed
BSD 2-Clause "Simplified" License
18 stars 10 forks source link

Upgrade bdv and imglib dependencies while keeping scijava and spark version. #15

Closed kgabor closed 1 year ago

kgabor commented 1 year ago
kgabor commented 1 year ago

In a 1-machine setup, the fatjar works for me with spark-submit, with the ancient gson version problem inherited from spark-2.4.7. Needs @trautmane 's workaround. (https://github.com/PreibischLab/BigStitcher-Spark/issues/11)

StephanPreibisch commented 1 year ago

@trautmane - could you please check this out?

StephanPreibisch commented 1 year ago

Excluding ijp-kheops bioformats from multiview-reconstruction makes total sense, only required for OpenSeaDragon export ...

StephanPreibisch commented 1 year ago

Note: I'll also make a new branch that uses the newest multiview-reconstruction 3.2.2 since it contains most of the code required in this spark-project ... I pushed everything upstream, also optimizing it at the same time

kgabor commented 1 year ago

@StephanPreibisch Thanks for merging these changes. I'm closing this PR then. When do you plan to merge overhaul-mvr-322 into main ?

I wanted to try upgrading Spark to 3.3.2 (this is the version run by Amazon EMR serverless).

kgabor commented 1 year ago

Superseded by #17.