Closed dpshepherd closed 2 years ago
Hi, I think its because you load from HDF5, which we didn't fix yet ... soon it'll work :)
No problem, just giving the information that Eric asked for. Happy to try again when appropriate.
Thanks!
Hi @dpshepherd - I pulled in changes from @tpietzsch that hopefully fix this problem for you. Can you build a 0.0.2-SNAPSHOT fatjar (-P fatjar
) from the main branch, remove the -XX:ActiveProcessorCount=1
hack, and re-run? Let us know how it goes. Thanks!
Hi @trautmane - we are still getting errors.
The fusion then starts, runs for awhile, throws another error, and then quietly exits.
Thanks @dpshepherd - can you post the log from the worker that ran task 8 (should contain the NPE you see in the driver log)? The worker logs usually are a stdout or stderr file depending upon your Spark setup.
Oops - just noticed you are running locally. I'm interested in getSparkJobSpimData2: returning ...
in the log(s). Do you see that anywhere?
I just added some friendlier error checking in the area where you are seeing NPEs: https://github.com/PreibischLab/BigStitcher-Spark/commit/3e0e6753b037a6eefe16d18634efd1850499d63e
Can you rebuild with the latest code, recreate the issue, and post/share the logs? Thanks!
Hi @dpshepherd, could you please send us the XML you are using? This might be happening because of missing views I suspect ...
It’s uploaded in the original post for this issue above.
I started the fusion again with logging and it hasn’t crashed yet. It did give the same errors in the beginning as I posted before.
Thanks! Ok, we could rule out the missing views issue ... will check with @trautmane in more detail what's going on ...
Here is the log for the fusion, which completed. There do appear to be errors in it.
A spot check on the output appears that it worked. I started creating a downsampled pyramid using the n5-spark
tools for viewing, since it is so large.
local-1642170932286.zip
Thanks @dpshepherd - your latest log shows that for many (maybe all?) of your 32 tasks, the first two attempts to load a setupImgLoader for setupId 0, timePointId 0 fails.
The third attempt then succeeds and processing continues.
I'll talk with @StephanPreibisch to get a better idea of what might be causing this. Thanks for your patience ...
@trautmane - no problem!
Let me know if there is anything we can do on our side. The drive that the data is stored on is network mounted via fiber connection - I wonder if this has anything to do with it? We've never seen a problem like this before using parallel processing in Python with Zarr stores on that drive or using BigStitcher (both through Fiji GUI and headless) on that drive.
I unfortunately have no experience developing in Java and this is the first time we have used Spark.
Ok, I can verify there are missing chunks (black boxes).
Hi @dpshepherd - I was able to recreate the latest setupImgLoader issue you encountered which allowed me to experiment with a couple of fix options. This commit fixed the issues for me. Can you rebuild with the latest code and let us know how it goes?
Thanks!
I rebuilt and started with logging. Peeking at the log I don't see the large number of errors at initialization. Will update once it finishes.
Hi @trautmane - this run finished without any errors in 18 hours! This is a massive speed-up from BigStitcher, the last time we tried to stitch this data on the same machine it was going to take ~336 hours (14 days). One observation is the RAM usage climbed up to a max of ~250 Gb. Is this expected? It is not a problem for us, because our server has a lot of ram - but may be important for others.
I am creating the downsampled pyramid for viewing and will post a screenshot to verify there are no missing blocks then.
Thanks!!!
Awesome, thanks so much @trautmane and @dpshepherd for pushing through :)
hi @StephanPreibisch and @trautmane
Appears successful with the n5-spark
n5-scale-pyramid.py
. We do get some file I/O errors when running the n5-spark
code,
but can't find any missing blocks. This is one channel (nuclei using DAPI) of a roughly 100 um thick brain biopsy that is >1 cm long on the long axis imaged with our Snouty-OPM. We'll start the other three channels fusing now.
Exciting and thanks again!!
Hi @trautmane,
As requested, here are the details on what we are running into trying to fuse a BDV file using
BigStitcher-Spark
. The plugin was built with the code changes on main, but notfix_bdv_n5
as I got a conflict when I tried to merge the two branches. I've attached the XML as well.It wasn't totally clear if the
extraJavaOptions
should be passed to thedriver
orexecutors
when in local mode, so we tried both. The same error as pasted here message pops up. We also tried allocating more RAM to the executors, same error message as pasted here pops up.The error usually occurs once >6,000 files within the N5 have been written. In this particular case, ~7,200 files were written.
Please let me know what other information I can provide.
Thanks! Doug
Linux version
```linux Linux qi2labserver 5.4.0-74-generic #83~18.04.1-Ubuntu SMP Tue May 11 16:01:00 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux ```Maven and java details
```linux Apache Maven 3.6.0 Maven home: /usr/share/maven Java version: 1.8.0_312, vendor: Private Build, runtime: /usr/lib/jvm/java-8-openjdk-amd64/jre Default locale: en_US, platform encoding: UTF-8 OS name: "linux", version: "5.4.0-74-generic", arch: "amd64", family: "unix" ```Spark details
```linux 22/01/04 11:54:33 WARN Utils: Your hostname, qi2labserver resolves to a loopback address: 127.0.1.1; using 10.206.25.77 instead (on interface enp5s0f0) 22/01/04 11:54:33 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 3.2.0 /_/ Using Scala version 2.12.15, OpenJDK 64-Bit Server VM, 1.8.0_312 Branch HEAD Compiled by user ubuntu on 2021-10-06T12:46:30Z Revision 5d45a415f3a29898d92380380cfd82bfc7f579ea Url https://github.com/apache/spark Type --help for more information. ```Call to Spark
```linux spark-submit --master local[32,8] --conf spark.driver.memory=100G --conf "spark.executor.extraJavaOptions=-XX:ActiveProcessorCount=1" --class net.preibisch.bigstitcher.spark.AffineFusion ~/Documents/github/BigStitcher-Spark/target/BigStitcher-Spark-0.0.1-SNAPSHOT.jar -x /mnt/opm2/20210924b/deskew_flatfield_output/bdv/AMC_cy7_test_bdv.xml -o /mnt/opm2/20210924b/n5/output.n5 -d /DAPI/s0 --channelId 0 --UINT16 --minIntensity 0 --maxIntensity 65535 ```Error message
```linux (Tue Jan 04 11:21:37 MST 2022): Requesting Img from ImgLoader (tp=0, setup=7), using level=0, [1.0 x 1.0 x 1.0] 22/01/04 11:21:37 ERROR Executor: Exception in task 28.0 in stage 0.0 (TID 28) java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:717) at net.imglib2.cache.queue.FetcherThreads.XML file: AMC_cy7_test_bdv.zip