search

leginon-org / leginon-redmine-archive

1 stars 0 forks source link

Images downloaded using dbemwiz are empty files #1331

Open leginonbot opened 6 months ago

leginonbot commented 6 months ago

Author Name: Neil Voss (@vosslab) Original Redmine Issue: 1331, https://emg.nysbc.org/redmine/issues/1331 Original Date: 2011-06-14

I was trying to download some images using dbemwiz. It is my first time using it and I was able to get it to download, but instead of downloading any images it was downloading blank files. Actually the files all contained the plain text:

string(2) "50"

I was trying with sessions, 11jan12b and 11jan06b. Amber/Anchi any ideas?

leginonbot commented 6 months ago

Original Redmine Comment Author Name: Amber Herold (Amber Herold) Original Date: 2011-06-17T15:31:39Z

Jim, Christopher and I have been unable to locate the source code for dbemwiz. Checked source control, and Denis and Eric's home directories. Do you have any idea where it might be?

Neil, we can put the images on a drive and send them to you if we don't get a solution today...probably easiest/fastest that way.

leginonbot commented 6 months ago

Original Redmine Comment Author Name: Amber Herold (Amber Herold) Original Date: 2011-06-17T16:15:35Z

Neil, there may be an issue with those data sets because they are on the external data drive /ami/exdata. You can try one of the other sets to see if they work for you. If you need those specific sets, we'll send them to you.

leginonbot commented 6 months ago

Original Redmine Comment Author Name: Jim Pulokas (Jim Pulokas) Original Date: 2011-06-17T18:13:31Z

I looked everywhere I could think of. All I found was an email dated May 15, 2010 from Denis where he says he is going to send us the source code. I do not have any follow-up email where the source code is actually sent, so maybe this only went directly to Eric or someone else. Can everyone check there old emails around that date to see if they have an attachment.

leginonbot commented 6 months ago

Original Redmine Comment Author Name: Neil Voss (@vosslab) Original Date: 2011-06-17T18:30:50Z

Thanks for looking at this. Sorry, I did not try another set until now, but sets on the /ami/dataXX drives work fine, so it is an issue with /ami/exdata.

I know Eric made several updates to the dbemwiz code, so it is unfortunate that you could not find it in his home directory. But it does work now.

I will get back to you on Monday on how to get my data, I may just ship you a drive.

Thanks guys!

leginonbot commented 6 months ago

Original Redmine Comment Author Name: Clint Potter (Clint Potter) Original Date: 2011-06-17T18:48:03Z

Here is source code from Denis email of 5/17/10

From: "Denis F." denis@nanoimagingservices.com Date: May 17, 2010 12:39:56 AM PDT To: Bridget Carragher bcarr@scripps.edu, erichou@scripps.edu Cc: Clint Potter cpotter@nanoimagingservices.com, Jim Pulokas pulokas@nanoimagingservices.com Subject: Re: Downloader

Please find attached the java source code. testwiz describes how to compile and run dbem wizard. The idea of this wizard is to use php to feed/stream data via http requests; no need to share directories /ami/dataXX online via httpd.conf, since PHP can already access the file system. This was a prototype, no heavy usage; most users ended up bringing their own hard-drives. You might want to use a servlet to talk to the java client for heavy usage, but I found handy to use the public website for authentication, etc, since the users can already view their data online.

Hope this helps

-Denis.

The files you want to look into are: ami/JgetURL.java ami/DownloadTask.java ami/LoginWizardPanel.java

The http request descriptions:

get summary info, su.php:

String link = "http://emg.nysbc.org/experiment/su.php?exp="+expId;

get a list of filenames, list.php:

String link = "http://emg.nysbc.org/experiment/list.php?exp="+expId;

get a list of filenames queried on preset, lq.php:

String link = "http://emg.nysbc.org/experiment/lq.php?exp="+expId+querystr;

get a list of experiments, le.php:

URL urlexp = new URL("http://emg.nysbc.org/experiment/le.php");

download data, dl.php:

in = jg.download(new URL("http://emg.nysbc.org/experiment/dl.php?exp="+expIdstr+"&file="+curfilestr));

login, lg.php:

URL urlexp = new URL("http://emg.nysbc.org/experiment/lg.php");

leginonbot commented 6 months ago

Original Redmine Comment Author Name: Amber Herold (Amber Herold) Original Date: 2011-06-17T18:50:39Z

Thanks Clint!

Neil, Christopher mounted exdata on the web server, so go ahead and give those datasets another try!

leginonbot commented 6 months ago

Original Redmine Comment Author Name: Neil Voss (@vosslab) Original Date: 2011-06-20T13:00:58Z

So, I downloaded 124GB over the weekend and all of them are coming up in EMAN v2 as invalid files. After investigating further I found using a hex editor that all the files started with @string(2) "50"\n@ then followed by the MRC file.

Using the command:

tail -c +16 11jan06b_00020gr_00019sq_00012hl_v01_00002en2.mrc > edit.mrc

should remove the header and make it a normal mrc file and it worked, but what a pain. Using Jim's hack from my Yahoo! email (not posted here for security reasons) it works fine with out hack, so there is a problem in dbemwiz.

System: Scientific Linux 6, Java 1.6, x86_64

leginonbot commented 6 months ago

Original Redmine Comment Author Name: Neil Voss (@vosslab) Original Date: 2011-07-06T10:34:37Z

Another bug is that the last four pixels at the end of the image file were not downloaded, so I had to edit them and included four blank pixels.

Nonetheless, I downloaded all 3925 images this way and despite the removal of the 'string(2) "50"\n' at the beginning and the missing pixels all the data is fine. But it is probably time to kill dbemwiz and come up with a new solution.