search

facebookresearch / fastMRI

A large-scale dataset of both raw MRI measurements and clinical MRI images.
https://fastmri.org
MIT License
1.29k stars 370 forks source link

Failed submissions to fastmri.org when using Google Drive #223

Closed mmuckley closed 1 year ago

mmuckley commented 2 years ago

This is something we identified in Discussion #220. It seems Google has changed their API recently so that our old code will fail to download a submission. Until I get some time to fix this, please submit to fastmri.org using Dropbox or an S3 bucket.

wdika commented 2 years ago

Following that, it seems like submitting using Dropbox also causes issues. I made a test submission yesterday using Dropbox, and it appears that the whole public leaderboard went down. @mmuckley you may want to take a look at that.

mmuckley commented 2 years ago

Okay I'm reaching out to a contractor that helps us run the site. Will try to follow-up when I can get something.

mmuckley commented 2 years ago

Hello @wdika, the site should be up and running now. Hoping to make an update so this particular error doesn't take it down again.

EDIT: I may have spoken too soon.

wdika commented 2 years ago

Hello @wdika, the site should be up and running now. Hoping to make an update so this particular error doesn't take it down again.

EDIT: I may have spoken too soon.

Hi @mmuckley, I think it's my fault again for the site being down. I noticed it was working like yesterday at some point and tried to resubmit. Then it went down again.

All I do is to set a dropbox link and change the last digit (on the URL) from 0 to 1 to enable writing permissions. The zip file is of the form mymodel.zip -> mymodel/multicoil_test/ . Btw I submitted on the Multi-Coil Brain track.

Maybe I should wait until any further notice from you to resubmit.

mmuckley commented 2 years ago

Hello @wdika, not sure it was you, but yes it was something with the database. I just made a tweak to the site and it should be more robust now. Leaderboard is up and you should be able to submit freely.

wdika commented 2 years ago

Hello @wdika, not sure it was you, but yes it was something with the database. I just made a tweak to the site and it should be more robust now. Leaderboard is up and you should be able to submit freely.

Hi, I can see that the leaderboard is up and running now but still I cannot properly submit. I can make a new submission but it doesn't show up in the leaderboard, neither on my account. On my account all the submissions are gone.

I even made a new account and made a new submission. I could see the submission was processing (on my personal board) and I got the email that it was successful. But it gone completely and it doesn't show up on the leaderboard at all.

mmuckley commented 2 years ago

Thanks @wdika for your assistance. I was able to identify the issue and you should see everything now.

wdika commented 2 years ago

Great @mmuckley, thanks for the update! You might remove the last models on the leaderboard since they are failed (as expected), because I cannot see them on my account.

mmuckley commented 2 years ago

@wdika I removed them. Let me know if I missed any.

wdika commented 2 years ago

@mmuckley I can submit now and see my submission but I think the targets have disappeared and the metrics are off. At first, I thought my reconstructions were faulty but as I realize now the targets are not there (left side). Can you please verify this? You may check the last model (xyz) on the brains leaderboard.

mmuckley commented 2 years ago

@wdika it looks like your images might be on the wrong scale?

The evaluation code for calculating metrics has not been changed for years and I would prefer not to alter it at all to maintain consistency. So there is some issue with how you formatted the data for your submission.

wdika commented 2 years ago

@wdika it looks like your images might be on the wrong scale?

The evaluation code for calculating metrics has not been changed for years and I would prefer not to alter it at all to maintain consistency. So there is some issue with how you formatted the data for your submission.

I made a few submissions, where I tested the scaling but none worked. Of course, it doesn't make sense to touch the evaluation code, but I am wondering why the targets' images show all black and the reconstructions look okay. Is this behavior expected?

mmuckley commented 2 years ago

No it's not expected. The images on the leaderboard are basically just a concatenation of your reconstruction with the target, so it looks like your reconstruction is several orders of magnitude off in scale. I would try to do some debugging with the validation data on your local machine to try to figure out the right scaling.

Sorry - don't think I can help more with this; it's not a bug with the site.

wdika commented 2 years ago

No it's not expected. The images on the leaderboard are basically just a concatenation of your reconstruction with the target, so it looks like your reconstruction is several orders of magnitude off in scale. I would try to do some debugging with the validation data on your local machine to try to figure out the right scaling.

Sorry - don't think I can help more with this; it's not a bug with the site.

Yeah, this is what I do. Thanks for all your help!

mmuckley commented 2 years ago

Resolved comments unrelated to issue title. If continued problems arise, please create a new issue.

Can-Zhao commented 1 year ago

Could I ask whether the issue is resolved? Thank you!

mmuckley commented 1 year ago

Hello @Can-Zhao it is not resolved, mostly because I'm working on several other projects and don't have time to fix it. I apologize for the inconvenience. Using an S3 bucket or Dropbox should still work.

mmuckley commented 1 year ago

Closing this as the website is no longer maintained by Meta.