Submission Failed due to error in evaluation pipeline

[jeromepatel]

Hi, My submission failed after a very long time. the error message I received is the following which suggests error inside evaluation code:

Traceback (most recent call last):
  File "/code/scripts/workers/submission_worker.py", line 500, in run_submission
    submission_metadata=submission_serializer.data,
  File "/tmp/tmpfotnqgdv/compute/challenge_data/challenge_2031/main.py", line 238, in evaluate
    challenge_metrics.evaluate_metrics(pred_parts_labels, pred_mats_labels,
UnboundLocalError: local variable 'pred_parts_labels' referenced before assignment

Also, is it possible to extend the deadline by a few days, since the issue was related to the evaluation pipeline?

Regards, Jyot

[HabibSlim]

Hey Jyot,

1) We've noticed the same bug with your submission and we're currently trying to fix it. We'll get back to you whenever we have a fix.

2) The deadline has been extended to June 10th.

Best, Habib

[HabibSlim]

@jeromepatel

The actual traceback for the error is not due to the error you're showing.

The issue with your submission is its format. The size of your HDF5 file is over 1.5GB. It should not exceed 100MB if you follow the instructions in our documentation.

Please make sure you're following the submission format we describe in our docpage:

https://3dcompat-dataset.org/doc/challenge.html

In general, your submission should not contain any float value, only integer predictions over the test or validation sets are accepted.

Please also make sure that you upload your submission as an .h5 file (instead of .hdf5, although this should not cause any errors in theory).

Let us know how we can help :)

Habib

[jeromepatel]

Thanks, Habib for your suggestion and help, will try to check my submission again. And a small request from my side, is it possible to open-source the model.py files for pre-trained models (since we only have access to the weights but not the architecture for those models, eg. PointNeXt)?

[HabibSlim]

No worries!

https://github.com/Vision-CAIR/3DCoMPaT-v2/tree/main/models/3D/PointNeXt

All the model files are in the PointNeXt folder, we also provide some example training scripts but you will have to adapt it to the 3D dataloaders we provide.

Best, Habib

[jeromepatel]

@HabibSlim Hello Habib, Can you please cancel my old submission which has been running for a long time, Unfortunately, I can not cancel it from my end. Update: The submission was stopped running somehow and I deleted it but my new submission is still in submitted status and not started.

Also, I could not find the model_name for segformer model for 2D segmentation trained models (that makes it perhaps impossible for me to load model as I don't know the architecture)

Thanks! Bests, Jyot