Closed mmuckley closed 1 year ago
Following that, it seems like submitting using Dropbox also causes issues. I made a test submission yesterday using Dropbox, and it appears that the whole public leaderboard went down. @mmuckley you may want to take a look at that.
Okay I'm reaching out to a contractor that helps us run the site. Will try to follow-up when I can get something.
Hello @wdika, the site should be up and running now. Hoping to make an update so this particular error doesn't take it down again.
EDIT: I may have spoken too soon.
Hello @wdika, the site should be up and running now. Hoping to make an update so this particular error doesn't take it down again.
EDIT: I may have spoken too soon.
Hi @mmuckley, I think it's my fault again for the site being down. I noticed it was working like yesterday at some point and tried to resubmit. Then it went down again.
All I do is to set a dropbox link and change the last digit (on the URL) from 0 to 1 to enable writing permissions. The zip file is of the form mymodel.zip -> mymodel/multicoil_test/ . Btw I submitted on the Multi-Coil Brain track.
Maybe I should wait until any further notice from you to resubmit.
Hello @wdika, not sure it was you, but yes it was something with the database. I just made a tweak to the site and it should be more robust now. Leaderboard is up and you should be able to submit freely.
Hello @wdika, not sure it was you, but yes it was something with the database. I just made a tweak to the site and it should be more robust now. Leaderboard is up and you should be able to submit freely.
Hi, I can see that the leaderboard is up and running now but still I cannot properly submit. I can make a new submission but it doesn't show up in the leaderboard, neither on my account. On my account all the submissions are gone.
I even made a new account and made a new submission. I could see the submission was processing (on my personal board) and I got the email that it was successful. But it gone completely and it doesn't show up on the leaderboard at all.
Thanks @wdika for your assistance. I was able to identify the issue and you should see everything now.
Great @mmuckley, thanks for the update! You might remove the last models on the leaderboard since they are failed (as expected), because I cannot see them on my account.
@wdika I removed them. Let me know if I missed any.
@mmuckley I can submit now and see my submission but I think the targets have disappeared and the metrics are off. At first, I thought my reconstructions were faulty but as I realize now the targets are not there (left side). Can you please verify this? You may check the last model (xyz) on the brains leaderboard.
@wdika it looks like your images might be on the wrong scale?
The evaluation code for calculating metrics has not been changed for years and I would prefer not to alter it at all to maintain consistency. So there is some issue with how you formatted the data for your submission.
@wdika it looks like your images might be on the wrong scale?
The evaluation code for calculating metrics has not been changed for years and I would prefer not to alter it at all to maintain consistency. So there is some issue with how you formatted the data for your submission.
I made a few submissions, where I tested the scaling but none worked. Of course, it doesn't make sense to touch the evaluation code, but I am wondering why the targets' images show all black and the reconstructions look okay. Is this behavior expected?
No it's not expected. The images on the leaderboard are basically just a concatenation of your reconstruction with the target, so it looks like your reconstruction is several orders of magnitude off in scale. I would try to do some debugging with the validation data on your local machine to try to figure out the right scaling.
Sorry - don't think I can help more with this; it's not a bug with the site.
No it's not expected. The images on the leaderboard are basically just a concatenation of your reconstruction with the target, so it looks like your reconstruction is several orders of magnitude off in scale. I would try to do some debugging with the validation data on your local machine to try to figure out the right scaling.
Sorry - don't think I can help more with this; it's not a bug with the site.
Yeah, this is what I do. Thanks for all your help!
Resolved comments unrelated to issue title. If continued problems arise, please create a new issue.
Could I ask whether the issue is resolved? Thank you!
Hello @Can-Zhao it is not resolved, mostly because I'm working on several other projects and don't have time to fix it. I apologize for the inconvenience. Using an S3 bucket or Dropbox should still work.
Closing this as the website is no longer maintained by Meta.
This is something we identified in Discussion #220. It seems Google has changed their API recently so that our old code will fail to download a submission. Until I get some time to fix this, please submit to fastmri.org using Dropbox or an S3 bucket.