Open leginonbot opened 7 months ago
Original Redmine Comment Author Name: Sargis Dallakyan (@dallakyan) Original Date: 2015-08-13T15:07:44Z
r19003 adds opencv-python. centosAutoInstallation.py exits with an error:
Could not connect: Access denied for user 'root'@'localhost' (using password: NO)
Working on fixing this...
Original Redmine Comment Author Name: Sargis Dallakyan (@dallakyan) Original Date: 2015-08-14T15:02:45Z
Adding the list of packages to be included in centosAutoInstallation.py
Original Redmine Comment Author Name: Alex Noble (@alexjnoble) Original Date: 2015-08-29T10:37:02Z
Hi Sargis,
Can you update us on the progress of getting Protomo auto-installed by default? Thanks, -Alex
Original Redmine Comment Author Name: Sargis Dallakyan (@dallakyan) Original Date: 2015-08-29T18:18:34Z
Hi Alex,
I'm committed r19105 that makes it possible for auto-installer to create databases and tables without root password for mysql. The good news that auto-installer now finishes successfully and it opens up working myamiweb and leginon after it's done.
On getting Protomo auto-installed by default, I have 80% of the code written. I hope to finish remaining 20% and commit it before the next Appion hangout.
Original Redmine Comment Author Name: Sargis Dallakyan (@dallakyan) Original Date: 2015-09-02T23:47:26Z
r19121 implements Protomo installer. I've tested this on one of our servers (fly) and it works as expected.
There are a couple of things related to opencv and mysql, that I can discuss with Anchi before closing this issue.
try:
import cv2
except:
import cv as cv2
* _mysqld_safe --skip-grant-tables &_ in the current centosAutoInstallation.py works fine and it allows creating databases and tables without root password, however, it won't survive a reboot. I'm thinking to change "auto-installation tool":http://emg.nysbc.org/redmine/projects/appion/wiki/Install_Appion_and_Leginon_using_the_auto-installation_tool_share wiki page and ask users to run "mysql_secure_installation":http://www.networkinghowtos.com/howto/secure-a-mysql-installation-using-mysql_secure_installation/ to set mysql root password and then update their sinedon and myamiweb config files.
Original Redmine Comment Author Name: Anchi Cheng (@anchi2c) Original Date: 2015-11-02T19:06:32Z
Carl tested the autoinstallor and then handed to me to testrun.
I got the following error while running a mock protomo2aligner.py :
sh: i3cut: command not found
... Alignment Correlation Peak Images and/or Videos could not be generated. Make sure i3, ffmpeg, and imagemagick are in your $PATH. Make sure that pyami and scipy are in your $PYTHONPATH.
Also got an error from the same script on FFTmpeg
FFmpeg version 0.6.5, Copyright (c) 2000-2010 the FFmpeg developers
built on Jan 29 2012 17:52:15 with gcc 4.4.5 20110214 (Red Hat 4.4.5-6)
configuration: --prefix=/usr --libdir=/usr/lib64 --shlibdir=/usr/lib64 --mandir=/usr/share/man --incdir=/usr/include --disable-avisynth --extra-cflags='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -fPIC' --enable-avfilter --enable-avfilter-lavf --enable-libdc1394 --enable-libdirac --enable-libfaac --enable-libfaad --enable-libfaadbin --enable-libgsm --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-librtmp --enable-libschroedinger --enable-libspeex --enable-libtheora --enable-libx264 --enable-gpl --enable-nonfree --enable-postproc --enable-pthreads --enable-shared --enable-swscale --enable-vdpau --enable-version3 --enable-x11grab
libavutil 50.15. 1 / 50.15. 1
libavcodec 52.72. 2 / 52.72. 2
libavformat 52.64. 2 / 52.64. 2
libavdevice 52. 2. 0 / 52. 2. 0
libavfilter 1.19. 0 / 1.19. 0
libswscale 0.11. 0 / 0.11. 0
libpostproc 51. 2. 0 / 51. 2. 0
Unrecognized option 'framerate'
Original Redmine Comment Author Name: Anchi Cheng (@anchi2c) Original Date: 2015-11-02T19:07:05Z
elevate it to urgent since we need to release 3.2
Original Redmine Comment Author Name: Alex Noble (@alexjnoble) Original Date: 2015-11-02T19:13:43Z
If i3cut isn't found, then i3 isn't installed or isn't in the path.
Here's my FFmpeg configuration, which works without issue:
configuration: --enable-gpl --enable-version3 --disable-shared --disable-debug --enable-runtime-cpudetect --enable-libmp3lame --enable-libx264 --enable-libx265 --enable-libwebp --enable-libspeex --enable-libvorbis --enable-libvpx --enable-libfreetype --enable-fontconfig --enable-libxvid --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libtheora --enable-libvo-aacenc --enable-libvo-amrwbenc --enable-gray --enable-libopenjpeg --enable-libopus --enable-libass --enable-gnutls --enable-libvidstab --cc=gcc-4.9
I'm using a 2015 ffmpeg version N-45094-g1932f7e-
Original Redmine Comment Author Name: Sargis Dallakyan (@dallakyan) Original Date: 2015-11-03T00:39:14Z
r19286 fixes i3cut and other issues. I use repoforge to install ffmpeg. Perhaps there are other repos that have FFmpeg compiled with the options Alex wrote, but for now this is good for shipping. Docker containers would make it easier to compile and distribute third party packages with the options we need.
Closing this issue for now; we can revisit this in the next release, if needed.
Original Redmine Comment Author Name: Carl Negro (@carl9384) Original Date: 2015-11-11T18:36:54Z
Alex,
Can you provide installation instructions for your working FFmpeg version? The autoinstaller currently uses the FFmpeg from the CentOS yum repository, which still causes the same framerate error Anchi mentioned as well as ```i3cut: could not load libi3tiffio.so, TiffioModule disabled
Original Redmine Comment Author Name: Alex Noble (@alexjnoble) Original Date: 2015-11-11T18:42:45Z
The i3cut error is not a problem. i3 has a list of image libraries that it tries to load in a particular order; if the first image library isn't present then it goes to the next one. That's all that error signifies - it found the next one and continued working.
It's been many months since I installed the ffmpeg I use so I don't remember what I did exactly... I'll install it again on my end and report back.
Original Redmine Comment Author Name: Alex Noble (@alexjnoble) Original Date: 2015-11-11T19:51:35Z
I use the static git version of ffmpeg:
http://johnvansickle.com/ffmpeg/
I just tried the latest 64bit build and it runs perfectly.
Original Redmine Comment Author Name: Anchi Cheng (@anchi2c) Original Date: 2015-11-11T19:58:46Z
Alex,
When protomo2align.py what files should we have and how big relative to the images in raw directory ? Our test run, apart from not getting media files had small files in croase_align and recon. I also only used 12 tilts (1 deg increment centered at 0). would that be a problem ?
Original Redmine Comment Author Name: Alex Noble (@alexjnoble) Original Date: 2015-11-11T20:58:25Z
The cache directory should be as large as the raw directory.
It shouldn't matter how many tilts you work with as long as it's more than maybe 3 or 5, I'm not sure.
Here is the complete terminal output for a Coarse Alignment run with all options as default (except I set frame_aligned=True): http://pastie.org/private/6hjmylrcctsuo8rezx7tya
Here is the corresponding log file: http://pastie.org/private/cd9ajzxxpw7vvy99szejq
Original Redmine Comment Author Name: Anchi Cheng (@anchi2c) Original Date: 2015-11-11T21:08:43Z
Looks like the output size is reasonable with our small tilt series. Now we are only waiting for a good fftmpeg installation through autoinstaller.
Original Redmine Comment Author Name: Carl Negro (@carl9384) Original Date: 2015-11-12T19:33:39Z
I rewrote the i3 paths in the autoinstaller so that protomo2aligner.py finishes, but throws the 'i3cut: could not load libi3tiffio.so, TiffioModule disabled'. Still working on this
Original Redmine Comment Author Name: Alex Noble (@alexjnoble) Original Date: 2015-11-12T19:46:23Z
If you can take this attached .img file and run the following commands:
i3cut -fmt mrc series000108_cor.img series000108_cor.mrc proc3d series000108_cor.mrc series000108_cor.mrc norm 3dmod series000108_cor.mrc
and you can see something, then nothing needs to be fixed. Even we get the "i3cut: could not load libi3tiffio.so, TiffioModule disabled" warning and we're using a Hanspeter installation!
Original Redmine Comment Author Name: Carl Negro (@carl9384) Original Date: 2015-11-14T16:37:15Z
Alex,
Here's the console output from your instructions:
[root@vagrant-centos65 local]# wget http://emg.nysbc.org/redmine/attachments/download/4677/series000108_cor.img --2015-11-14 16:31:41-- http://emg.nysbc.org/redmine/attachments/download/4677/series000108_cor.img Resolving emg.nysbc.org... 172.85.57.157 Connecting to emg.nysbc.org|172.85.57.157|:80... connected. HTTP request sent, awaiting response... 200 OK Length: unspecified [text/plain] Saving to: “series000108_cor.img.1”
[ <=> ] 3,788,920 1.47M/s in 2.5s
2015-11-14 16:31:44 (1.47 MB/s) - “series000108_cor.img.1” saved [3788920]
[root@vagrant-centos65 local]# i3cut -fmt mrc series000108_cor.img series000108_cor.mrc i3cut: could not load libi3tiffio.so, TiffioModule disabled [root@vagrant-centos65 local]# proc3d series000108_cor.mrc series000108_cor.mrc norm EMAN 1.9 Cluster ($Date: 2009/02/18 05:12:22 $). Run 'proc3d help' for detailed help. Original image : 160x160x37 Mean=3.2e-05 Sigma=0.00427 Min=-0.0205 Max=0.0394 Sampling : 1 A/pixel Origin = 0,0,0 Final image : 160x160x37 Mean=4.38e-09 Sigma=1 Sampling : 1 A/pixel Origin = 0,0,0 [root@vagrant-centos65 local]# 3dmod series000108_cor.mrc -bash: 3dmod: command not found [root@vagrant-centos65 local]#
Is this the output you were anticipating?
Also, could you post the contents of 'echo $PATH', 'echo $PYTHONPATH', and 'echo $LD_LIBRARY_PATH' on your machine? Thanks
Original Redmine Comment Author Name: Alex Noble (@alexjnoble) Original Date: 2015-11-14T16:50:07Z
Yes it works fine, you just don't have 3dmod (ie. IMOD) installed.
i3cut -fmt mrc input.img output.mrc just takes the Protomo-specific img file format and converts it to mrc. proc3d then normalizes it so that you can open it up and see it in 3dmod: http://bio3d.colorado.edu/imod/doc/3dmodguide.html#ZapWindow
But you don't have 3dmod installed (which is fine). Open it up in any EM image viewer that you prefer to convince yourself.
Original Redmine Comment Author Name: Alex Noble (@alexjnoble) Original Date: 2015-11-14T16:54:24Z
bash-4.2$ echo $PATH /panfs/storage.local/imb/home/ajn10d/bin/screen-4.2.1:/panfs/storage.local/imb/home/ajn10d/bin/automatch_v0.3.1_withdata/bin:/panfs/storage.local/imb/home/ajn10d/bin/Komodo-Edit-8.5.4/bin:/panfs/storage.local/imb/stagg/software/rhel61/usr/local/IMOD/bin:/panfs/storage.local/imb/stagg/software/myami_python/bin:/panfs/storage.local/imb/home/ajn10d/myami3/redux/bin:/panfs/storage.local/imb/home/ajn10d/myami3/appion/bin:/panfs/storage.local/imb/stagg/software/rhel61/usr/local/bin:/panfs/storage.local/imb/home/ajn10d/bin/ffmpeg-git-20150311-64bit-static:/panfs/storage.local/imb/home/sstagg/Komodo-Edit-8/bin:/panfs/storage.local/imb/stagg/software/rhel61/usr/local/Xmipp/bin:/panfs/storage.local/imb/stagg/software/rhel61/usr/local/spider18.15/spider/bin/:/panfs/storage.local/imb/stagg/software/rhel61/usr/local/simple_linux_120521/apps:/panfs/storage.local/imb/stagg/software/rhel61/usr/local/simple_linux_120521/bin:/panfs/storage.local/imb/stagg/software/rhel61/usr/local/relion-1.3/bin:/panfs/storage.local/imb/scratch/software/i3-0.9.6/bin:/panfs/storage.local/imb/scratch/software/i3-0.9.6/bin/linux/x86-64:/panfs/storage.local/imb/scratch/software/protomo-2.4.1/bin:/panfs/storage.local/imb/scratch/software/protomo-2.4.1/bin/linux/x86-64:/panfs/storage.local/imb/home/sstagg/hsource_rss_20100325:/panfs/storage.local/imb/stagg/software/rhel61/usr/local/grigorieff/frealign_v9.11/bin:/panfs/storage.local/imb/home/sstagg/software/rhel61/usr/local/EMAN/bin:/panfs/storage.local/imb/home/sstagg/software/rhel61/usr/local/EMAN2/bin:/opt/hpc/gnu/xalt/bin:/panfs/storage.local/opt/hpc/gnu/openmpi/bin:/opt/hpc/gnu/openmpi/bin:/panfs/storage.local/opt/hpc/gnu/bin:/opt/hpc/gnu/bin:/opt/matlab/2014b/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/panfs/storage.local/imb/home/ajn10d/bin:/panfs/storage.local/imb/home/ajn10d/bin/dynamo:/panfs/storage.local/opt/hpc/gnu/bin:/opt/hpc/gnu/bin:/opt/matlab/2014b/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/panfs/storage.local/imb/home/ajn10d/bin:/opt/hpc/gnu:/panfs/storage.local/opt/hpc/gnu/bin:/opt/hpc/gnu/bin:/opt/matlab/2014b/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/panfs/storage.local/imb/home/ajn10d/bin:/panfs/storage.local/imb/home/ajn10d/bin/dynamo-v-1.1.62_MCR-8.2.0_GLNXA64/matlab/bin:/panfs/storage.local/imb/home/ajn10d/bin/dynamo-v-1.1.62_MCR-8.2.0_GLNXA64/matlab/src:/panfs/storage.local/imb/home/ajn10d/bin/dynamo-v-1.1.62_MCR-8.2.0_GLNXA64/cuda/bin:/panfs/storage.local/imb/home/ajn10d/bin/dynamo-v-1.1.62_MCR-8.2.0_GLNXA64/mpi:/panfs/storage.local/imb/home/ajn10d/bin/dynamo-v-1.1.62_MCR-8.2.0_GLNXA64/examples:/panfs/storage.local/imb/home/ajn10d/bin/dynamo-v-1.1.62_MCR-8.2.0_GLNXA64/doc
bash-4.2$ echo $PYTHONPATH /panfs/storage.local/imb/home/ajn10d/myami3/appion:/panfs/storage.local/imb/home/ajn10d/myami3/leginon:/panfs/storage.local/imb/home/ajn10d/myami3:/panfs/storage.local/imb/stagg/software/rhel61/usr/local/Xmipp/lib:/panfs/storage.local/imb/scratch/software/protomo-2.4.1/lib/linux/x86-64:/panfs/storage.local/imb/home/sstagg/software/rhel61/usr/local/EMAN/lib:/panfs/storage.local/imb/stagg/software/rhel61/usr/local/bin:/panfs/storage.local/imb/home/sstagg/software/rhel61/usr/local/EMAN2/bin:/panfs/storage.local/imb/home/sstagg/software/rhel61/usr/local/EMAN2/lib:/opt/hpc/gnu/xalt/site:/opt/hpc/gnu/xalt/libexec:/usr/lib64/python2.6/site-packages/openmpi
bash-4.2$ echo $LD_LIBRARY_PATH /usr/local/cuda-7.0/lib64:/panfs/storage.local/imb/stagg/software/rhel61/usr/local/IMOD/lib:/panfs/storage.local/imb/stagg/software/rhel61/usr/local/IMOD/lib:/panfs/storage.local/imb/stagg/software/rhel61/usr/local/Xmipp/lib:/panfs/storage.local/imb/stagg/software/rhel61/usr/local/relion-1.3/lib:/panfs/storage.local/imb/scratch/software/protomo-2.4.1/lib/linux/x86-64:/panfs/storage.local/imb/scratch/software/lib/linux/x86-64:/panfs/storage.local/imb/stagg/software/rhel61/usr/local/lib:/panfs/storage.local/imb/home/sstagg/software/rhel61/usr/local/EMAN/lib:/panfs/storage.local/imb/home/sstagg/software/rhel61/usr/local/EMAN2/lib:/panfs/storage.local/opt/hpc/gnu/openmpi/lib64:/opt/hpc/gnu/openmpi/lib64:/usr/lib64/openmpi/lib:/panfs/storage.local/opt/hpc/gnu/lib64:/opt/hpc/gnu/lib64:/opt/hpc/gnu/lib64/netcdf-3:/panfs/storage.local/imb/home/ajn10d/bin/MCR_82/v82/runtime/glnxa64:/panfs/storage.local/imb/home/ajn10d/bin/MCR_82/v82/bin/glnxa64:/panfs/storage.local/imb/home/ajn10d/bin/MCR_82/v82/sys/os/glnxa64:/panfs/storage.local/imb/home/ajn10d/bin/MCR_82/v82/sys/java/jre/glnxa64/jre/lib/amd64/native_threads:/panfs/storage.local/imb/home/ajn10d/bin/MCR_82/v82/sys/java/jre/glnxa64/jre/lib/amd64/server:/panfs/storage.local/imb/home/ajn10d/bin/MCR_82/v82/sys/java/jre/glnxa64/jre/lib/amd64
Original Redmine Comment Author Name: Sargis Dallakyan (@dallakyan) Original Date: 2015-11-19T23:47:26Z
r19387 merges protomo related r19374 through r19383 from trunk to 3.2 brunch.
Author Name: Sargis Dallakyan (@dallakyan) Original Redmine Issue: 3343, https://emg.nysbc.org/redmine/issues/3343 Original Date: 2015-08-06 Original Assignee: Alex Noble
Will need to modify "auto-installation tool":http://emg.nysbc.org/redmine/projects/appion/wiki/Install_Appion_and_Leginon_using_the_auto-installation_tool_share to include "protomo":http://emg.nysbc.org/redmine/attachments/download/4147/protomo-2.4.1.zip and opencv-python for the next release (3.2).