search

leginon-org / leginon-redmine-archive

1 stars 0 forks source link

Add protomo and opencv-python to centosAutoInstallation.py #3343

Open leginonbot opened 8 months ago

leginonbot commented 8 months ago

Author Name: Sargis Dallakyan (@dallakyan) Original Redmine Issue: 3343, https://emg.nysbc.org/redmine/issues/3343 Original Date: 2015-08-06 Original Assignee: Alex Noble

Will need to modify "auto-installation tool":http://emg.nysbc.org/redmine/projects/appion/wiki/Install_Appion_and_Leginon_using_the_auto-installation_tool_share to include "protomo":http://emg.nysbc.org/redmine/attachments/download/4147/protomo-2.4.1.zip and opencv-python for the next release (3.2).

leginonbot commented 8 months ago

Original Redmine Comment Author Name: Sargis Dallakyan (@dallakyan) Original Date: 2015-08-13T15:07:44Z

r19003 adds opencv-python. centosAutoInstallation.py exits with an error:

Could not connect: Access denied for user 'root'@'localhost' (using password: NO)

Working on fixing this...

leginonbot commented 8 months ago

Original Redmine Comment Author Name: Sargis Dallakyan (@dallakyan) Original Date: 2015-08-14T15:02:45Z

Adding the list of packages to be included in centosAutoInstallation.py

leginonbot commented 8 months ago

Original Redmine Comment Author Name: Alex Noble (@alexjnoble) Original Date: 2015-08-29T10:37:02Z

Hi Sargis,

Can you update us on the progress of getting Protomo auto-installed by default? Thanks, -Alex

leginonbot commented 8 months ago

Original Redmine Comment Author Name: Sargis Dallakyan (@dallakyan) Original Date: 2015-08-29T18:18:34Z

Hi Alex,

I'm committed r19105 that makes it possible for auto-installer to create databases and tables without root password for mysql. The good news that auto-installer now finishes successfully and it opens up working myamiweb and leginon after it's done.

On getting Protomo auto-installed by default, I have 80% of the code written. I hope to finish remaining 20% and commit it before the next Appion hangout.

leginonbot commented 8 months ago

Original Redmine Comment Author Name: Sargis Dallakyan (@dallakyan) Original Date: 2015-09-02T23:47:26Z

r19121 implements Protomo installer. I've tested this on one of our servers (fly) and it works as expected.

There are a couple of things related to opencv and mysql, that I can discuss with Anchi before closing this issue.



* _mysqld_safe --skip-grant-tables &_ in the current centosAutoInstallation.py works fine and it allows creating databases and tables without root password, however, it won't survive a reboot. I'm thinking to change "auto-installation tool":http://emg.nysbc.org/redmine/projects/appion/wiki/Install_Appion_and_Leginon_using_the_auto-installation_tool_share wiki page and ask users to run  "mysql_secure_installation":http://www.networkinghowtos.com/howto/secure-a-mysql-installation-using-mysql_secure_installation/ to set mysql root password and then update their sinedon and myamiweb config files.
leginonbot commented 8 months ago

Original Redmine Comment Author Name: Anchi Cheng (@anchi2c) Original Date: 2015-11-02T19:06:32Z

Carl tested the autoinstallor and then handed to me to testrun.

I got the following error while running a mock protomo2aligner.py :

sh: i3cut: command not found
 ... Alignment Correlation Peak Images and/or Videos could not be generated. Make sure i3, ffmpeg, and imagemagick are in your $PATH. Make sure that pyami and scipy are in your $PYTHONPATH.

Also got an error from the same script on FFTmpeg

FFmpeg version 0.6.5, Copyright (c) 2000-2010 the FFmpeg developers
  built on Jan 29 2012 17:52:15 with gcc 4.4.5 20110214 (Red Hat 4.4.5-6)
  configuration: --prefix=/usr --libdir=/usr/lib64 --shlibdir=/usr/lib64 --mandir=/usr/share/man --incdir=/usr/include --disable-avisynth --extra-cflags='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -fPIC' --enable-avfilter --enable-avfilter-lavf --enable-libdc1394 --enable-libdirac --enable-libfaac --enable-libfaad --enable-libfaadbin --enable-libgsm --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-librtmp --enable-libschroedinger --enable-libspeex --enable-libtheora --enable-libx264 --enable-gpl --enable-nonfree --enable-postproc --enable-pthreads --enable-shared --enable-swscale --enable-vdpau --enable-version3 --enable-x11grab
  libavutil     50.15. 1 / 50.15. 1
  libavcodec    52.72. 2 / 52.72. 2
  libavformat   52.64. 2 / 52.64. 2
  libavdevice   52. 2. 0 / 52. 2. 0
  libavfilter    1.19. 0 /  1.19. 0
  libswscale     0.11. 0 /  0.11. 0
  libpostproc   51. 2. 0 / 51. 2. 0
Unrecognized option 'framerate'
leginonbot commented 8 months ago

Original Redmine Comment Author Name: Anchi Cheng (@anchi2c) Original Date: 2015-11-02T19:07:05Z

elevate it to urgent since we need to release 3.2

leginonbot commented 8 months ago

Original Redmine Comment Author Name: Alex Noble (@alexjnoble) Original Date: 2015-11-02T19:13:43Z

If i3cut isn't found, then i3 isn't installed or isn't in the path.

Here's my FFmpeg configuration, which works without issue:

configuration: --enable-gpl --enable-version3 --disable-shared --disable-debug --enable-runtime-cpudetect --enable-libmp3lame --enable-libx264 --enable-libx265 --enable-libwebp --enable-libspeex --enable-libvorbis --enable-libvpx --enable-libfreetype --enable-fontconfig --enable-libxvid --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libtheora --enable-libvo-aacenc --enable-libvo-amrwbenc --enable-gray --enable-libopenjpeg --enable-libopus --enable-libass --enable-gnutls --enable-libvidstab --cc=gcc-4.9

I'm using a 2015 ffmpeg version N-45094-g1932f7e-

leginonbot commented 8 months ago

Original Redmine Comment Author Name: Sargis Dallakyan (@dallakyan) Original Date: 2015-11-03T00:39:14Z

r19286 fixes i3cut and other issues. I use repoforge to install ffmpeg. Perhaps there are other repos that have FFmpeg compiled with the options Alex wrote, but for now this is good for shipping. Docker containers would make it easier to compile and distribute third party packages with the options we need.

Closing this issue for now; we can revisit this in the next release, if needed.

leginonbot commented 8 months ago

Original Redmine Comment Author Name: Carl Negro (@carl9384) Original Date: 2015-11-11T18:36:54Z

Alex,

Can you provide installation instructions for your working FFmpeg version? The autoinstaller currently uses the FFmpeg from the CentOS yum repository, which still causes the same framerate error Anchi mentioned as well as ```i3cut: could not load libi3tiffio.so, TiffioModule disabled

leginonbot commented 8 months ago

Original Redmine Comment Author Name: Alex Noble (@alexjnoble) Original Date: 2015-11-11T18:42:45Z

The i3cut error is not a problem. i3 has a list of image libraries that it tries to load in a particular order; if the first image library isn't present then it goes to the next one. That's all that error signifies - it found the next one and continued working.

It's been many months since I installed the ffmpeg I use so I don't remember what I did exactly... I'll install it again on my end and report back.

leginonbot commented 8 months ago

Original Redmine Comment Author Name: Alex Noble (@alexjnoble) Original Date: 2015-11-11T19:51:35Z

I use the static git version of ffmpeg:

http://johnvansickle.com/ffmpeg/

I just tried the latest 64bit build and it runs perfectly.

leginonbot commented 8 months ago

Original Redmine Comment Author Name: Anchi Cheng (@anchi2c) Original Date: 2015-11-11T19:58:46Z

Alex,

When protomo2align.py what files should we have and how big relative to the images in raw directory ? Our test run, apart from not getting media files had small files in croase_align and recon. I also only used 12 tilts (1 deg increment centered at 0). would that be a problem ?

leginonbot commented 8 months ago

Original Redmine Comment Author Name: Alex Noble (@alexjnoble) Original Date: 2015-11-11T20:58:25Z

The cache directory should be as large as the raw directory.

It shouldn't matter how many tilts you work with as long as it's more than maybe 3 or 5, I'm not sure.

Here is the complete terminal output for a Coarse Alignment run with all options as default (except I set frame_aligned=True): http://pastie.org/private/6hjmylrcctsuo8rezx7tya

Here is the corresponding log file: http://pastie.org/private/cd9ajzxxpw7vvy99szejq

leginonbot commented 8 months ago

Original Redmine Comment Author Name: Anchi Cheng (@anchi2c) Original Date: 2015-11-11T21:08:43Z

Looks like the output size is reasonable with our small tilt series. Now we are only waiting for a good fftmpeg installation through autoinstaller.

leginonbot commented 8 months ago

Original Redmine Comment Author Name: Carl Negro (@carl9384) Original Date: 2015-11-12T19:33:39Z

I rewrote the i3 paths in the autoinstaller so that protomo2aligner.py finishes, but throws the 'i3cut: could not load libi3tiffio.so, TiffioModule disabled'. Still working on this

leginonbot commented 8 months ago

Original Redmine Comment Author Name: Alex Noble (@alexjnoble) Original Date: 2015-11-12T19:46:23Z

If you can take this attached .img file and run the following commands:

i3cut -fmt mrc series000108_cor.img series000108_cor.mrc proc3d series000108_cor.mrc series000108_cor.mrc norm 3dmod series000108_cor.mrc

and you can see something, then nothing needs to be fixed. Even we get the "i3cut: could not load libi3tiffio.so, TiffioModule disabled" warning and we're using a Hanspeter installation!

leginonbot commented 8 months ago

Original Redmine Comment Author Name: Carl Negro (@carl9384) Original Date: 2015-11-14T16:37:15Z

Alex,

Here's the console output from your instructions:

[root@vagrant-centos65 local]# wget http://emg.nysbc.org/redmine/attachments/download/4677/series000108_cor.img --2015-11-14 16:31:41-- http://emg.nysbc.org/redmine/attachments/download/4677/series000108_cor.img Resolving emg.nysbc.org... 172.85.57.157 Connecting to emg.nysbc.org|172.85.57.157|:80... connected. HTTP request sent, awaiting response... 200 OK Length: unspecified [text/plain] Saving to: “series000108_cor.img.1”

[             <=>                       ] 3,788,920   1.47M/s   in 2.5s

2015-11-14 16:31:44 (1.47 MB/s) - “series000108_cor.img.1” saved [3788920]

[root@vagrant-centos65 local]# i3cut -fmt mrc series000108_cor.img series000108_cor.mrc i3cut: could not load libi3tiffio.so, TiffioModule disabled [root@vagrant-centos65 local]# proc3d series000108_cor.mrc series000108_cor.mrc norm EMAN 1.9 Cluster ($Date: 2009/02/18 05:12:22 $). Run 'proc3d help' for detailed help. Original image : 160x160x37 Mean=3.2e-05 Sigma=0.00427 Min=-0.0205 Max=0.0394 Sampling : 1 A/pixel Origin = 0,0,0 Final image : 160x160x37 Mean=4.38e-09 Sigma=1 Sampling : 1 A/pixel Origin = 0,0,0 [root@vagrant-centos65 local]# 3dmod series000108_cor.mrc -bash: 3dmod: command not found [root@vagrant-centos65 local]#

Is this the output you were anticipating?

Also, could you post the contents of 'echo $PATH', 'echo $PYTHONPATH', and 'echo $LD_LIBRARY_PATH' on your machine? Thanks

leginonbot commented 8 months ago

Original Redmine Comment Author Name: Alex Noble (@alexjnoble) Original Date: 2015-11-14T16:50:07Z

Yes it works fine, you just don't have 3dmod (ie. IMOD) installed.

i3cut -fmt mrc input.img output.mrc just takes the Protomo-specific img file format and converts it to mrc. proc3d then normalizes it so that you can open it up and see it in 3dmod: http://bio3d.colorado.edu/imod/doc/3dmodguide.html#ZapWindow

But you don't have 3dmod installed (which is fine). Open it up in any EM image viewer that you prefer to convince yourself.

leginonbot commented 8 months ago

Original Redmine Comment Author Name: Alex Noble (@alexjnoble) Original Date: 2015-11-14T16:54:24Z

bash-4.2$ echo $PATH /panfs/storage.local/imb/home/ajn10d/bin/screen-4.2.1:/panfs/storage.local/imb/home/ajn10d/bin/automatch_v0.3.1_withdata/bin:/panfs/storage.local/imb/home/ajn10d/bin/Komodo-Edit-8.5.4/bin:/panfs/storage.local/imb/stagg/software/rhel61/usr/local/IMOD/bin:/panfs/storage.local/imb/stagg/software/myami_python/bin:/panfs/storage.local/imb/home/ajn10d/myami3/redux/bin:/panfs/storage.local/imb/home/ajn10d/myami3/appion/bin:/panfs/storage.local/imb/stagg/software/rhel61/usr/local/bin:/panfs/storage.local/imb/home/ajn10d/bin/ffmpeg-git-20150311-64bit-static:/panfs/storage.local/imb/home/sstagg/Komodo-Edit-8/bin:/panfs/storage.local/imb/stagg/software/rhel61/usr/local/Xmipp/bin:/panfs/storage.local/imb/stagg/software/rhel61/usr/local/spider18.15/spider/bin/:/panfs/storage.local/imb/stagg/software/rhel61/usr/local/simple_linux_120521/apps:/panfs/storage.local/imb/stagg/software/rhel61/usr/local/simple_linux_120521/bin:/panfs/storage.local/imb/stagg/software/rhel61/usr/local/relion-1.3/bin:/panfs/storage.local/imb/scratch/software/i3-0.9.6/bin:/panfs/storage.local/imb/scratch/software/i3-0.9.6/bin/linux/x86-64:/panfs/storage.local/imb/scratch/software/protomo-2.4.1/bin:/panfs/storage.local/imb/scratch/software/protomo-2.4.1/bin/linux/x86-64:/panfs/storage.local/imb/home/sstagg/hsource_rss_20100325:/panfs/storage.local/imb/stagg/software/rhel61/usr/local/grigorieff/frealign_v9.11/bin:/panfs/storage.local/imb/home/sstagg/software/rhel61/usr/local/EMAN/bin:/panfs/storage.local/imb/home/sstagg/software/rhel61/usr/local/EMAN2/bin:/opt/hpc/gnu/xalt/bin:/panfs/storage.local/opt/hpc/gnu/openmpi/bin:/opt/hpc/gnu/openmpi/bin:/panfs/storage.local/opt/hpc/gnu/bin:/opt/hpc/gnu/bin:/opt/matlab/2014b/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/panfs/storage.local/imb/home/ajn10d/bin:/panfs/storage.local/imb/home/ajn10d/bin/dynamo:/panfs/storage.local/opt/hpc/gnu/bin:/opt/hpc/gnu/bin:/opt/matlab/2014b/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/panfs/storage.local/imb/home/ajn10d/bin:/opt/hpc/gnu:/panfs/storage.local/opt/hpc/gnu/bin:/opt/hpc/gnu/bin:/opt/matlab/2014b/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/panfs/storage.local/imb/home/ajn10d/bin:/panfs/storage.local/imb/home/ajn10d/bin/dynamo-v-1.1.62_MCR-8.2.0_GLNXA64/matlab/bin:/panfs/storage.local/imb/home/ajn10d/bin/dynamo-v-1.1.62_MCR-8.2.0_GLNXA64/matlab/src:/panfs/storage.local/imb/home/ajn10d/bin/dynamo-v-1.1.62_MCR-8.2.0_GLNXA64/cuda/bin:/panfs/storage.local/imb/home/ajn10d/bin/dynamo-v-1.1.62_MCR-8.2.0_GLNXA64/mpi:/panfs/storage.local/imb/home/ajn10d/bin/dynamo-v-1.1.62_MCR-8.2.0_GLNXA64/examples:/panfs/storage.local/imb/home/ajn10d/bin/dynamo-v-1.1.62_MCR-8.2.0_GLNXA64/doc

bash-4.2$ echo $PYTHONPATH /panfs/storage.local/imb/home/ajn10d/myami3/appion:/panfs/storage.local/imb/home/ajn10d/myami3/leginon:/panfs/storage.local/imb/home/ajn10d/myami3:/panfs/storage.local/imb/stagg/software/rhel61/usr/local/Xmipp/lib:/panfs/storage.local/imb/scratch/software/protomo-2.4.1/lib/linux/x86-64:/panfs/storage.local/imb/home/sstagg/software/rhel61/usr/local/EMAN/lib:/panfs/storage.local/imb/stagg/software/rhel61/usr/local/bin:/panfs/storage.local/imb/home/sstagg/software/rhel61/usr/local/EMAN2/bin:/panfs/storage.local/imb/home/sstagg/software/rhel61/usr/local/EMAN2/lib:/opt/hpc/gnu/xalt/site:/opt/hpc/gnu/xalt/libexec:/usr/lib64/python2.6/site-packages/openmpi

bash-4.2$ echo $LD_LIBRARY_PATH /usr/local/cuda-7.0/lib64:/panfs/storage.local/imb/stagg/software/rhel61/usr/local/IMOD/lib:/panfs/storage.local/imb/stagg/software/rhel61/usr/local/IMOD/lib:/panfs/storage.local/imb/stagg/software/rhel61/usr/local/Xmipp/lib:/panfs/storage.local/imb/stagg/software/rhel61/usr/local/relion-1.3/lib:/panfs/storage.local/imb/scratch/software/protomo-2.4.1/lib/linux/x86-64:/panfs/storage.local/imb/scratch/software/lib/linux/x86-64:/panfs/storage.local/imb/stagg/software/rhel61/usr/local/lib:/panfs/storage.local/imb/home/sstagg/software/rhel61/usr/local/EMAN/lib:/panfs/storage.local/imb/home/sstagg/software/rhel61/usr/local/EMAN2/lib:/panfs/storage.local/opt/hpc/gnu/openmpi/lib64:/opt/hpc/gnu/openmpi/lib64:/usr/lib64/openmpi/lib:/panfs/storage.local/opt/hpc/gnu/lib64:/opt/hpc/gnu/lib64:/opt/hpc/gnu/lib64/netcdf-3:/panfs/storage.local/imb/home/ajn10d/bin/MCR_82/v82/runtime/glnxa64:/panfs/storage.local/imb/home/ajn10d/bin/MCR_82/v82/bin/glnxa64:/panfs/storage.local/imb/home/ajn10d/bin/MCR_82/v82/sys/os/glnxa64:/panfs/storage.local/imb/home/ajn10d/bin/MCR_82/v82/sys/java/jre/glnxa64/jre/lib/amd64/native_threads:/panfs/storage.local/imb/home/ajn10d/bin/MCR_82/v82/sys/java/jre/glnxa64/jre/lib/amd64/server:/panfs/storage.local/imb/home/ajn10d/bin/MCR_82/v82/sys/java/jre/glnxa64/jre/lib/amd64

leginonbot commented 8 months ago

Original Redmine Comment Author Name: Sargis Dallakyan (@dallakyan) Original Date: 2015-11-19T23:47:26Z

r19387 merges protomo related r19374 through r19383 from trunk to 3.2 brunch.