search

TGAC / KAT

The K-mer Analysis Toolkit (KAT) contains a number of tools that analyse and compare K-mer spectra.
http://www.earlham.ac.uk/kat-tools
GNU General Public License v3.0
208 stars 52 forks source link

scripts/setup.py does not respect DESTDIR variable when installing file (to a temporary location) #94

Closed mmokrejs closed 3 years ago

mmokrejs commented 6 years ago

Unfortunately the scripts try to fidlle with the site-wide directories (escaping the building sandbox subdirectories).This was denied by the build system:

make[3]: Entering directory '/scratch/var/tmp/portage/sci-biology/kat-9999/work/kat-9999/scripts'
export PYTHONPATH=/usr/lib/python3/site-packages && \
mkdir -p /usr/lib/python3/site-packages/ && \
python3 setup.py install --prefix=/usr --record install_files.txt
 * ACCESS DENIED:  mkdir:        /usr/lib64/python3
mkdir: cannot create directory '/usr/lib/python3': Permission denied
make[3]: *** [Makefile:490: install-exec-hook] Error 1
make[3]: Leaving directory '/scratch/var/tmp/portage/sci-biology/kat-9999/work/kat-9999/scripts'
make[2]: *** [Makefile:432: install-exec-am] Error 2
make[2]: Leaving directory '/scratch/var/tmp/portage/sci-biology/kat-9999/work/kat-9999/scripts'
make[1]: *** [Makefile:380: install-am] Error 2
make[1]: Leaving directory '/scratch/var/tmp/portage/sci-biology/kat-9999/work/kat-9999/scripts'
make: *** [Makefile:470: install-recursive] Error 1
 * ERROR: sci-biology/kat-9999::science failed (install phase):
 *   emake failed
 * 
 * If you need support, post the output of `emerge --info '=sci-biology/kat-9999::science'`,
 * the complete build log and the output of `emerge -pqv '=sci-biology/kat-9999::science'`.
 * The complete build log is located at '/scratch/var/tmp/portage/sci-biology/kat-9999/temp/build.log'.
 * The ebuild environment file is located at '/scratch/var/tmp/portage/sci-biology/kat-9999/temp/environment'.
 * Working directory: '/scratch/var/tmp/portage/sci-biology/kat-9999/work/kat-9999'
 * S: '/scratch/var/tmp/portage/sci-biology/kat-9999/work/kat-9999'
 * --------------------------- ACCESS VIOLATION SUMMARY ---------------------------
 * LOG FILE: "/var/log/sandbox/sandbox-27386.log"
 * 
VERSION 1.0
FORMAT: F - Function called
FORMAT: S - Access Status
FORMAT: P - Path as passed to function
FORMAT: A - Absolute Path (not canonical)
FORMAT: R - Canonical Path
FORMAT: C - Command Line

F: mkdir
S: deny
P: python3
A: /usr/lib64/python3
R: /usr/lib64/python3
C: mkdir -p /usr/lib/python3 site-packages/ 
 * --------------------------------------------------------------------------------

Hmm, the command should have been python3 setup.py install --prefix="${ED}"/usr --record install_files.txt. Somehow the setup.py does not respect DESTDIR variable like other python tools. The DESTDIR should have expanded into "${ED}"/usr.

A bit above in the log one can see other parts of KAT building respected the DESTDIR properly:

make[2]: Entering directory '/scratch/var/tmp/portage/sci-biology/kat-9999/work/kat-9999/src'
 /bin/mkdir -p '/scratch/var/tmp/portage/sci-biology/kat-9999/image//usr/bin'

build.log.txt

maplesond commented 6 years ago

Strange. I'll take a look at the build.log.txt file to try and figure out what is happening here. Definitely, if you issued a prefix to the configuration script this should be reflected in the installation directory for these python packages. It also looks a bit strange in that it hasn't correct detected your python version. Normally, if no prefix is issued it should try to install to: /usr/local/lib/python3.X/site-packages.

maplesond commented 6 years ago

Yes, there seems to be some issue setting the correct python version. In you configure output there is this message: configure: Environment variable 'PYTHON_VERSION' set to: 3 This should be set to 3.5. Can you also also type: grep PYTHON config.log and send me the results please?

mmokrejs commented 6 years ago

OK, I switched to

-    # PYTHON_VERSION=3 econf ${myconf[@]}
+    econf ${myconf[@]}

and the error is now:

Making install in scripts
make[1]: Entering directory '/scratch/var/tmp/portage/sci-biology/kat-8888/work/kat-8888/scripts'
make[2]: Entering directory '/scratch/var/tmp/portage/sci-biology/kat-8888/work/kat-8888/scripts'
make  install-exec-hook
make[3]: Entering directory '/scratch/var/tmp/portage/sci-biology/kat-8888/work/kat-8888/scripts'
export PYTHONPATH=/usr/lib/python3.5/site-packages && \
mkdir -p /usr/lib/python3.5/site-packages/ && \
python3 setup.py install --prefix=/usr --record install_files.txt
running install
 * ACCESS DENIED:  open_wr:      /usr/lib64/python3.5/site-packages/test-easy-install-7340.write-test
error: can't create or remove files in install directory

The following error occurred while trying to add or remove files in the
installation directory:

    [Errno 13] Permission denied: '/usr/lib64/python3.5/site-packages/test-easy-install-7340.write-test'

The installation directory you specified (via --install-dir, --prefix, or
the distutils default setting) was:

    /usr/lib64/python3.5/site-packages/

Perhaps your account does not have write access to this directory?  If the
installation directory is a system-owned directory, you may need to sign in
as the administrator or "root" account.  If you do not have administrative
access to this machine, you may wish to choose a different installation
directory, preferably one that is listed in your PYTHONPATH environment
variable.

For information on other options, you may wish to consult the
documentation at:

  https://setuptools.readthedocs.io/en/latest/easy_install.html

Please make the appropriate changes for your system and try again.

make[3]: *** [Makefile:490: install-exec-hook] Error 1
make[3]: Leaving directory '/scratch/var/tmp/portage/sci-biology/kat-8888/work/kat-8888/scripts'
make[2]: *** [Makefile:432: install-exec-am] Error 2
make[2]: Leaving directory '/scratch/var/tmp/portage/sci-biology/kat-8888/work/kat-8888/scripts'
make[1]: *** [Makefile:380: install-am] Error 2
make[1]: Leaving directory '/scratch/var/tmp/portage/sci-biology/kat-8888/work/kat-8888/scripts'
make: *** [Makefile:470: install-recursive] Error 1
 * ERROR: sci-biology/kat-8888::science failed (install phase):
 *   emake failed
 * --------------------------- ACCESS VIOLATION SUMMARY ---------------------------
 * LOG FILE: "/var/log/sandbox/sandbox-6958.log"
 * 
VERSION 1.0
FORMAT: F - Function called
FORMAT: S - Access Status
FORMAT: P - Path as passed to function
FORMAT: A - Absolute Path (not canonical)
FORMAT: R - Canonical Path
FORMAT: C - Command Line

F: open_wr
S: deny
P: /usr/lib64/python3.5/site-packages/test-easy-install-7340.write-test
A: /usr/lib64/python3.5/site-packages/test-easy-install-7340.write-test
R: /usr/lib64/python3.5/site-packages/test-easy-install-7340.write-test
C: /usr/lib/python-exec/python3.5/python3 setup.py install --prefix=/usr --record install_files.txt 
 * --------------------------------------------------------------------------------
$ grep python config.log 
PATH: /usr/lib/portage/python2.7/ebuild-helpers/xattr
PATH: /usr/lib/portage/python2.7/ebuild-helpers
Configured with: /scratch/var/tmp/portage/sys-devel/gcc-7.3.0-r1/work/gcc-7.3.0/configure --host=x86_64-pc-linux-gnu --build=x86_64-pc-linux-gnu --prefix=/usr --bindir=/usr/x86_64-pc-linux-gnu/gcc-bin/7.3.0 --includedir=/usr/lib/gcc/x86_64-pc-linux-gnu/7.3.0/include --datadir=/usr/share/gcc-data/x86_64-pc-linux-gnu/7.3.0 --mandir=/usr/share/gcc-data/x86_64-pc-linux-gnu/7.3.0/man --infodir=/usr/share/gcc-data/x86_64-pc-linux-gnu/7.3.0/info --with-gxx-include-dir=/usr/lib/gcc/x86_64-pc-linux-gnu/7.3.0/include/g++-v7 --with-python-dir=/share/gcc-data/x86_64-pc-linux-gnu/7.3.0/python --enable-languages=c,c++,objc,fortran --enable-obsolete --enable-secureplt --disable-werror --with-system-zlib --enable-nls --without-included-gettext --enable-checking=release --with-bugurl=https://bugs.gentoo.org/ --with-pkgversion='Gentoo 7.3.0-r1 p1.1' --disable-esp --enable-libstdcxx-time --enable-shared --enable-threads=posix --enable-__cxa_atexit --enable-clocale=gnu --enable-multilib --with-multilib-list=m32,m64 --disable-altivec --disable-fixed-point --enable-targets=all --disable-libgcj --enable-libgomp --disable-libmudflap --disable-libssp --disable-libcilkrts --disable-libmpx --enable-vtable-verify --enable-libvtv --enable-lto --without-isl --enable-libsanitizer --enable-default-pie --enable-default-ssp
Configured with: /scratch/var/tmp/portage/sys-devel/gcc-7.3.0-r1/work/gcc-7.3.0/configure --host=x86_64-pc-linux-gnu --build=x86_64-pc-linux-gnu --prefix=/usr --bindir=/usr/x86_64-pc-linux-gnu/gcc-bin/7.3.0 --includedir=/usr/lib/gcc/x86_64-pc-linux-gnu/7.3.0/include --datadir=/usr/share/gcc-data/x86_64-pc-linux-gnu/7.3.0 --mandir=/usr/share/gcc-data/x86_64-pc-linux-gnu/7.3.0/man --infodir=/usr/share/gcc-data/x86_64-pc-linux-gnu/7.3.0/info --with-gxx-include-dir=/usr/lib/gcc/x86_64-pc-linux-gnu/7.3.0/include/g++-v7 --with-python-dir=/share/gcc-data/x86_64-pc-linux-gnu/7.3.0/python --enable-languages=c,c++,objc,fortran --enable-obsolete --enable-secureplt --disable-werror --with-system-zlib --enable-nls --without-included-gettext --enable-checking=release --with-bugurl=https://bugs.gentoo.org/ --with-pkgversion='Gentoo 7.3.0-r1 p1.1' --disable-esp --enable-libstdcxx-time --enable-shared --enable-threads=posix --enable-__cxa_atexit --enable-clocale=gnu --enable-multilib --with-multilib-list=m32,m64 --disable-altivec --disable-fixed-point --enable-targets=all --disable-libgcj --enable-libgomp --disable-libmudflap --disable-libssp --disable-libcilkrts --disable-libmpx --enable-vtable-verify --enable-libvtv --enable-lto --without-isl --enable-libsanitizer --enable-default-pie --enable-default-ssp
configure:3848: result: /usr/lib/portage/python2.7/ebuild-helpers/xattr/install -c
Configured with: /scratch/var/tmp/portage/sys-devel/gcc-7.3.0-r1/work/gcc-7.3.0/configure --host=x86_64-pc-linux-gnu --build=x86_64-pc-linux-gnu --prefix=/usr --bindir=/usr/x86_64-pc-linux-gnu/gcc-bin/7.3.0 --includedir=/usr/lib/gcc/x86_64-pc-linux-gnu/7.3.0/include --datadir=/usr/share/gcc-data/x86_64-pc-linux-gnu/7.3.0 --mandir=/usr/share/gcc-data/x86_64-pc-linux-gnu/7.3.0/man --infodir=/usr/share/gcc-data/x86_64-pc-linux-gnu/7.3.0/info --with-gxx-include-dir=/usr/lib/gcc/x86_64-pc-linux-gnu/7.3.0/include/g++-v7 --with-python-dir=/share/gcc-data/x86_64-pc-linux-gnu/7.3.0/python --enable-languages=c,c++,objc,fortran --enable-obsolete --enable-secureplt --disable-werror --with-system-zlib --enable-nls --without-included-gettext --enable-checking=release --with-bugurl=https://bugs.gentoo.org/ --with-pkgversion='Gentoo 7.3.0-r1 p1.1' --disable-esp --enable-libstdcxx-time --enable-shared --enable-threads=posix --enable-__cxa_atexit --enable-clocale=gnu --enable-multilib --with-multilib-list=m32,m64 --disable-altivec --disable-fixed-point --enable-targets=all --disable-libgcj --enable-libgomp --disable-libmudflap --disable-libssp --disable-libcilkrts --disable-libmpx --enable-vtable-verify --enable-libvtv --enable-lto --without-isl --enable-libsanitizer --enable-default-pie --enable-default-ssp
configure:18234: checking for python3
configure:18252: found /usr/bin/python3
configure:18264: result: /usr/bin/python3
configure:18300: Found python version: 3.5.5
configure:18411: result: -I/usr/include/python3.5m
configure:18507: result: -L/usr/lib64 -lpython3.5m
configure:18521: result: /usr/lib64/python3.5/site-packages
configure:18533: checking python extra libraries
configure:18547: checking python extra linking flags
configure:18561: checking consistency of all components of python development environment
configure:18589: x86_64-pc-linux-gnu-gcc -o conftest   -I/usr/include/python3.5m -Wl,-O1 -Wl,--as-needed -Xlinker -export-dynamic conftest.c  -L/usr/lib64 -lpython3.5m -lpthread -ldl  -lutil -lm -lpthread -ldl  -lutil -lm >&5
configure:18671: checking python3 module: numpy
configure:18706: checking python3 module: matplotlib
configure:18741: checking python3 module: scipy
configure:18776: checking python3 module: tabulate
ac_cv_path_PYTHON=/usr/bin/python3
ac_cv_path_install='/usr/lib/portage/python2.7/ebuild-helpers/xattr/install -c'
AM_CPPFLAGS='-I/usr/include -I/usr/include/python3.5m'
AM_LIBS='-pthread -lz -lrt -lpthread -ldl  -lutil -lm -L/usr/lib64 -lpython3.5m'
PYTHON='/usr/bin/python3'
PYTHON_CPPFLAGS='-I/usr/include/python3.5m'
PYTHON_LIBS='-L/usr/lib64 -lpython3.5m'
PYTHON_SITE_PKG='/usr/lib64/python3.5/site-packages'
#define PYTHON_INT_PATH "/usr/bin/python3"
#define PYTHON_INTERP_SITE_PKGS "/usr/lib64/python3.5/site-packages"
$

build.log.txt

mmokrejs commented 6 years ago

I think the problem is that you run python3 setup.py install --prefix=$(prefix) --record install_files.txt on your own in scripts/Makefile. This should be run by the building environment. What can I drop from scripts/Makefile.am?

maplesond commented 6 years ago

So is /usr/lib64/python3.5/site-packages/ where your system python3 packages are kept? I'm not really sure what's happening on your system, or why it's even looking in /usr/lib64. AFAIK that's nothing I've done. Probably best to look at the ./scripts/Makefile.am install hooks to see if you can identify the cause.

One other thing to try. You can avoid installing the python packages by adding --disable-pykat-install to the configure command. I'm not certain this will work in your case though.

maplesond commented 6 years ago

Ah, I think we cross posted. So you can also use --disable-pykat in the configuration file too, this effectively avoids the ./scripts/Makefile, which should allow you to get past this error but this will remove some useful functionality from KAT.

mmokrejs commented 6 years ago

I went with

--- scripts/Makefile.am.ori 2018-04-22 16:02:12.829046527 +0200
+++ scripts/Makefile.am 2018-04-22 16:05:31.444532061 +0200
@@ -1,27 +1,6 @@
 AUTOMAKE_OPTIONS = subdir-objects

-if MAKE_PYKAT_INSTALL
-
-install-exec-hook:
-   export PYTHONPATH=$(prefix)/lib/python$(PYTHON_VERSION)/site-packages && \
-   mkdir -p $(prefix)/lib/python$(PYTHON_VERSION)/site-packages/ && \
-   python3 setup.py install --prefix=$(prefix) --record install_files.txt
-
-uninstall-hook:
-   cat install_files.txt | xargs rm -rf
-
-else
-
-install-exec-hook:
-   mkdir -p $(prefix)/lib/python$(PYTHON_VERSION)/local/ && \
-   cp -r kat $(prefix)/lib/python$(PYTHON_VERSION)/local
-
-uninstall-hook:
-   rm -rf $(prefix)/lib/python$(PYTHON_VERSION)/local
-
-endif
-
 EXTRA_DIST = \
    kat/distanalysis.py \
    kat/plot/__init__.py \

Although I enter the ./scripts/ subdirectory and get setup.py called properly I am getting:

$ /usr/bin/kat
../lib/include/kat/kat_fs.hpp(167): Throw in function kat::KatFS::KatFS(const char*)
Dynamic exception type: boost::exception_detail::clone_impl<kat::FileSystemException>
std::exception::what: std::exception
[kat::FileSystemError*] = Could not find KAT scripts at the expected installed location: /usr/lib64/python3.5/local
$

Looks it is anyway looking for the files in the wrong place. I assume /usr/lib64/python3.5/site-packages/kat will be created once I get the setup.py called by Gentoo build system.

One more issue. The make process should die upon an error. Anyway, what was the underlying error with the pdf file?

Making install in doc
make[1]: Entering directory '/scratch/var/tmp/portage/sci-biology/kat-8888/work/kat-8888/doc'
test -s build && test -s build/man && /usr/bin/install -c -m 644 build/man/kat.1 /scratch/var/tmp/portage/sci-biology/kat-8888/image//usr/share/man/man1/kat.1
test -s build && test -s build/latex && test -s build/latex/kat.pdf && /usr/bin/install -c -m 644 build/latex/kat.pdf /scratch/var/tmp/portage/sci-biology/kat-8888/image//usr/share/doc/kat-8888/kat.pdf;
make[1]: [Makefile:273: install-pdf] Error 1 (ignored)
test -s build && test -s build/html && mkdir -p /scratch/var/tmp/portage/sci-biology/kat-8888/image//usr/share/doc/kat-8888/html && cp -rf build/html /scratch/var/tmp/portage/sci-biology/kat-8888/image//usr/share/doc/kat-8888/html && chmod 755 -R /scratch/var/tmp/portage/sci-biology/kat-8888/image//usr/share/doc/kat-8888/html
make[1]: Leaving directory '/scratch/var/tmp/portage/sci-biology/kat-8888/work/kat-8888/doc'

Few comments to what I ended up installed. I see an extra `html/' directory level here:

/usr/share/doc/kat-8888
/usr/share/doc/kat-8888/AUTHORS.bz2
/usr/share/doc/kat-8888/NEWS.bz2
/usr/share/doc/kat-8888/README.md.bz2
/usr/share/doc/kat-8888/html
/usr/share/doc/kat-8888/html/html
/usr/share/doc/kat-8888/html/html/.buildinfo
/usr/share/doc/kat-8888/html/html/_images
/usr/share/doc/kat-8888/html/html/_images/ccoli_comp.png
/usr/share/doc/kat-8888/html/html/_images/ccoli_gcp.png
/usr/share/doc/kat-8888/html/html/_images/ccoli_hist.png
...

Likewise, is the kat-2.4.1 really needed/helpful?

/usr/include/kat-2.4.1
/usr/include/kat-2.4.1/kat
/usr/include/kat-2.4.1/kat/comp_counters.hpp
/usr/include/kat-2.4.1/kat/distance_metrics.hpp
/usr/include/kat-2.4.1/kat/input_handler.hpp
/usr/include/kat-2.4.1/kat/jellyfish_helper.hpp
/usr/include/kat-2.4.1/kat/kat_fs.hpp
/usr/include/kat-2.4.1/kat/matrix_metadata_extractor.hpp
/usr/include/kat-2.4.1/kat/sparse_matrix.hpp
/usr/include/kat-2.4.1/kat/spectra_helper.hpp
/usr/include/kat-2.4.1/kat/str_utils.hpp

build.log.txt

mmokrejs commented 6 years ago

Yes, there is no doc/build/latex/ at all so build/latex/kat.pdf neither.

maplesond commented 6 years ago

Yes, that was because it is probably a bit much to ask users to have latex installed. However, if it is installed then it's still nice to generate the pdf.

I'll take a look at the other issues you've raised later.

mmokrejs commented 6 years ago

But where is the *.tex file? Anyway, I do have latex installed:

$ latex
This is pdfTeX, Version 3.14159265-2.6-1.40.18 (TeX Live 2017) (preloaded format=latex)
 restricted \write18 enabled.
**
! End of file on the terminal... why?
maplesond commented 6 years ago

Ahh ok I remember now. Yes, it's not the detection of latex that triggers this. If the user generate the pdf, via:

cd doc && make latexpdf

then when installed to the system it looks for the .pdf. However, quite likely the user hasn't done this hence the check in case it doesn't exist.

The .tex files get generated automatically from the .rst files in the doc/src directory via sphinx (if you have it installed). So .rst -> .tex -> .pdf. Also .rst -> .html and a bundle of other doc types if requested.

It's probably easiest not to support installation of the .pdf production though and leave it to the user to generate and copy where they like. If this is a problem I can remove this as it's no big loss.

mmokrejs commented 6 years ago

OK, so the Makefile.am rules should ensure cd doc && make latexpdf is executed. I am not asking for removal of the pdf-generating code, just asking so that:

  1. make exits upon an error
  2. that necessary rules get called before build/latex/kat.pdf is to be copied elsewhere
  3. proper check for latexmk exists: 
    sphinx-build -b latex -d build/doctrees   -c source ../doc/source build/latex
Running Sphinx v1.6.7
making output directory...
loading pickled environment... done
building [mo]: targets for 0 po files that are out of date
building [latex]: all documents
updating environment: 0 added, 0 changed, 0 removed
looking for now-outdated files... none found
processing kat.tex...index installation using kmer walkthrough faq 
resolving references...
writing... done
copying images... [100%] images/real_r1_v_r2.png                                                                                                                                                                                            
copying TeX support files...
done
build succeeded.
Running LaTeX files through pdflatex...
make -C build/latex all-pdf
make[1]: Entering directory '/scratch/var/tmp/portage/sci-biology/kat-8888/work/kat-8888/doc/build/latex'
latexmk -pdf -dvi- -ps-  'kat.tex'
make[1]: latexmk: Command not found
make[1]: *** [Makefile:33: kat.pdf] Error 127
make[1]: Leaving directory '/scratch/var/tmp/portage/sci-biology/kat-8888/work/kat-8888/doc/build/latex'
make: *** [Makefile:180: latexpdf] Error 2
* ERROR: sci-biology/kat-8888::science failed (compile phase):
*   (no error message)
* 
* Call stack:
*     ebuild.sh, line 124:  Called src_compile
*   environment, line 4882:  Called die
* The specific snippet of code:
*       cd doc && make latexpdf || die

I am still not done with with setup.py part either, I will need somebody else to take a look at it.

maplesond commented 6 years ago

Yeah, it's just easier to remove the part of the makefile that will install the PDF. A full LaTeX build that you'll need to create the PDF is collosal and I don't want users to need to install it. The docs will still be built and installed as html and man pages. I'll remove this in the next release.

maplesond commented 6 years ago

With regards to the other issues you raised here:

mmokrejs commented 6 years ago

I did not propose dropping the PDF manual rendering. I have latex, I installed latexmk, you just need to check for their presence and exit upon an error. Why discarding your work?

I thinkl the cp -rf build/html /scratch/var/tmp/portage/sci-biology/kat-8888/image//usr/share/doc/kat-8888/html introduces the extra level.

But kat doesn't seem to be made to instal multiple versions at the same time. So I wonder I could have more than just /usr/include/kat-2.4.1/kat.

mmokrejs commented 6 years ago

Here is what I get after the proper setup.py call through Gentoo build system.

build.log.txt

# equery files kat
 * Searching for kat ...
 * Contents of sci-biology/kat-2.4.1:
/usr
/usr/bin
/usr/bin/kat
/usr/bin/kat_distanalysis -> ../lib/python-exec/python-exec2
/usr/bin/kat_jellyfish
/usr/bin/kat_plot_cold -> ../lib/python-exec/python-exec2
/usr/bin/kat_plot_density -> ../lib/python-exec/python-exec2
/usr/bin/kat_plot_profile -> ../lib/python-exec/python-exec2
/usr/bin/kat_plot_spectra_cn -> ../lib/python-exec/python-exec2
/usr/bin/kat_plot_spectra_hist -> ../lib/python-exec/python-exec2
/usr/bin/kat_plot_spectra_mx -> ../lib/python-exec/python-exec2
/usr/include
/usr/include/kat-2.4.1
/usr/include/kat-2.4.1/kat
/usr/include/kat-2.4.1/kat/comp_counters.hpp
/usr/include/kat-2.4.1/kat/distance_metrics.hpp
/usr/include/kat-2.4.1/kat/input_handler.hpp
/usr/include/kat-2.4.1/kat/jellyfish_helper.hpp
/usr/include/kat-2.4.1/kat/kat_fs.hpp
/usr/include/kat-2.4.1/kat/matrix_metadata_extractor.hpp
/usr/include/kat-2.4.1/kat/sparse_matrix.hpp
/usr/include/kat-2.4.1/kat/spectra_helper.hpp
/usr/include/kat-2.4.1/kat/str_utils.hpp
/usr/include/kat_jellyfish-2.4.0
/usr/include/kat_jellyfish-2.4.0/jellyfish
/usr/include/kat_jellyfish-2.4.0/jellyfish/allocators_mmap.hpp
/usr/include/kat_jellyfish-2.4.0/jellyfish/atomic_bits_array.hpp
/usr/include/kat_jellyfish-2.4.0/jellyfish/atomic_field.hpp
/usr/include/kat_jellyfish-2.4.0/jellyfish/atomic_gcc.hpp
/usr/include/kat_jellyfish-2.4.0/jellyfish/backtrace.hpp
/usr/include/kat_jellyfish-2.4.0/jellyfish/binary_dumper.hpp
/usr/include/kat_jellyfish-2.4.0/jellyfish/bloom_common.hpp
/usr/include/kat_jellyfish-2.4.0/jellyfish/bloom_counter2.hpp
/usr/include/kat_jellyfish-2.4.0/jellyfish/bloom_filter.hpp
/usr/include/kat_jellyfish-2.4.0/jellyfish/circular_buffer.hpp
/usr/include/kat_jellyfish-2.4.0/jellyfish/compare_and_swap.hpp
/usr/include/kat_jellyfish-2.4.0/jellyfish/cooperative_pool.hpp
/usr/include/kat_jellyfish-2.4.0/jellyfish/cooperative_pool2.hpp
/usr/include/kat_jellyfish-2.4.0/jellyfish/cpp_array.hpp
/usr/include/kat_jellyfish-2.4.0/jellyfish/divisor.hpp
/usr/include/kat_jellyfish-2.4.0/jellyfish/dumper.hpp
/usr/include/kat_jellyfish-2.4.0/jellyfish/err.hpp
/usr/include/kat_jellyfish-2.4.0/jellyfish/file_header.hpp
/usr/include/kat_jellyfish-2.4.0/jellyfish/generator_manager.hpp
/usr/include/kat_jellyfish-2.4.0/jellyfish/generic_file_header.hpp
/usr/include/kat_jellyfish-2.4.0/jellyfish/gzstream.hpp
/usr/include/kat_jellyfish-2.4.0/jellyfish/hash_counter.hpp
/usr/include/kat_jellyfish-2.4.0/jellyfish/int128.hpp
/usr/include/kat_jellyfish-2.4.0/jellyfish/jellyfish.hpp
/usr/include/kat_jellyfish-2.4.0/jellyfish/json.h
/usr/include/kat_jellyfish-2.4.0/jellyfish/large_hash_array.hpp
/usr/include/kat_jellyfish-2.4.0/jellyfish/large_hash_iterator.hpp
/usr/include/kat_jellyfish-2.4.0/jellyfish/locks_pthread.hpp
/usr/include/kat_jellyfish-2.4.0/jellyfish/mapped_file.hpp
/usr/include/kat_jellyfish-2.4.0/jellyfish/mer_dna.hpp
/usr/include/kat_jellyfish-2.4.0/jellyfish/mer_dna_bloom_counter.hpp
/usr/include/kat_jellyfish-2.4.0/jellyfish/mer_heap.hpp
/usr/include/kat_jellyfish-2.4.0/jellyfish/mer_iterator.hpp
/usr/include/kat_jellyfish-2.4.0/jellyfish/mer_overlap_sequence_parser.hpp
/usr/include/kat_jellyfish-2.4.0/jellyfish/mer_qual_iterator.hpp
/usr/include/kat_jellyfish-2.4.0/jellyfish/misc.hpp
/usr/include/kat_jellyfish-2.4.0/jellyfish/offsets_key_value.hpp
/usr/include/kat_jellyfish-2.4.0/jellyfish/rectangular_binary_matrix.hpp
/usr/include/kat_jellyfish-2.4.0/jellyfish/simple_circular_buffer.hpp
/usr/include/kat_jellyfish-2.4.0/jellyfish/sorted_dumper.hpp
/usr/include/kat_jellyfish-2.4.0/jellyfish/stdio_filebuf.hpp
/usr/include/kat_jellyfish-2.4.0/jellyfish/storage.hpp
/usr/include/kat_jellyfish-2.4.0/jellyfish/stream_iterator.hpp
/usr/include/kat_jellyfish-2.4.0/jellyfish/stream_manager.hpp
/usr/include/kat_jellyfish-2.4.0/jellyfish/text_dumper.hpp
/usr/include/kat_jellyfish-2.4.0/jellyfish/thread_exec.hpp
/usr/include/kat_jellyfish-2.4.0/jellyfish/time.hpp
/usr/include/kat_jellyfish-2.4.0/jellyfish/token_ring.hpp
/usr/include/kat_jellyfish-2.4.0/jellyfish/whole_sequence_parser.hpp
/usr/lib
/usr/lib/python-exec
/usr/lib/python-exec/python3.5
/usr/lib/python-exec/python3.5/kat_distanalysis
/usr/lib/python-exec/python3.5/kat_plot_cold
/usr/lib/python-exec/python3.5/kat_plot_density
/usr/lib/python-exec/python3.5/kat_plot_profile
/usr/lib/python-exec/python3.5/kat_plot_spectra_cn
/usr/lib/python-exec/python3.5/kat_plot_spectra_hist
/usr/lib/python-exec/python3.5/kat_plot_spectra_mx
/usr/lib64
/usr/lib64/libkat.a
/usr/lib64/libkat.la
/usr/lib64/libkat.so -> libkat.so.2.0.4
/usr/lib64/libkat.so.2 -> libkat.so.2.0.4
/usr/lib64/libkat.so.2.0.4
/usr/lib64/libkat_jellyfish.a
/usr/lib64/libkat_jellyfish.la
/usr/lib64/libkat_jellyfish.so -> libkat_jellyfish.so.2.0.0
/usr/lib64/libkat_jellyfish.so.2 -> libkat_jellyfish.so.2.0.0
/usr/lib64/libkat_jellyfish.so.2.0.0
/usr/lib64/pkgconfig
/usr/lib64/pkgconfig/kat.pc
/usr/lib64/pkgconfig/kat_jellyfish.pc
/usr/lib64/python3.5
/usr/lib64/python3.5/site-packages
/usr/lib64/python3.5/site-packages/kat
/usr/lib64/python3.5/site-packages/kat-2.4.1-py3.5.egg-info
/usr/lib64/python3.5/site-packages/kat-2.4.1-py3.5.egg-info/PKG-INFO
/usr/lib64/python3.5/site-packages/kat-2.4.1-py3.5.egg-info/SOURCES.txt
/usr/lib64/python3.5/site-packages/kat-2.4.1-py3.5.egg-info/dependency_links.txt
/usr/lib64/python3.5/site-packages/kat-2.4.1-py3.5.egg-info/entry_points.txt
/usr/lib64/python3.5/site-packages/kat-2.4.1-py3.5.egg-info/not-zip-safe
/usr/lib64/python3.5/site-packages/kat-2.4.1-py3.5.egg-info/requires.txt
/usr/lib64/python3.5/site-packages/kat-2.4.1-py3.5.egg-info/top_level.txt
/usr/lib64/python3.5/site-packages/kat/__init__.py
/usr/lib64/python3.5/site-packages/kat/__pycache__
/usr/lib64/python3.5/site-packages/kat/__pycache__/__init__.cpython-35.opt-1.pyc
/usr/lib64/python3.5/site-packages/kat/__pycache__/__init__.cpython-35.opt-2.pyc
/usr/lib64/python3.5/site-packages/kat/__pycache__/__init__.cpython-35.pyc
/usr/lib64/python3.5/site-packages/kat/__pycache__/distanalysis.cpython-35.opt-1.pyc
/usr/lib64/python3.5/site-packages/kat/__pycache__/distanalysis.cpython-35.opt-2.pyc
/usr/lib64/python3.5/site-packages/kat/__pycache__/distanalysis.cpython-35.pyc
/usr/lib64/python3.5/site-packages/kat/__pycache__/peak.cpython-35.opt-1.pyc
/usr/lib64/python3.5/site-packages/kat/__pycache__/peak.cpython-35.opt-2.pyc
/usr/lib64/python3.5/site-packages/kat/__pycache__/peak.cpython-35.pyc
/usr/lib64/python3.5/site-packages/kat/__pycache__/spectra.cpython-35.opt-1.pyc
/usr/lib64/python3.5/site-packages/kat/__pycache__/spectra.cpython-35.opt-2.pyc
/usr/lib64/python3.5/site-packages/kat/__pycache__/spectra.cpython-35.pyc
/usr/lib64/python3.5/site-packages/kat/distanalysis.py
/usr/lib64/python3.5/site-packages/kat/peak.py
/usr/lib64/python3.5/site-packages/kat/plot
/usr/lib64/python3.5/site-packages/kat/plot/__init__.py
/usr/lib64/python3.5/site-packages/kat/plot/__pycache__
/usr/lib64/python3.5/site-packages/kat/plot/__pycache__/__init__.cpython-35.opt-1.pyc
/usr/lib64/python3.5/site-packages/kat/plot/__pycache__/__init__.cpython-35.opt-2.pyc
/usr/lib64/python3.5/site-packages/kat/plot/__pycache__/__init__.cpython-35.pyc
/usr/lib64/python3.5/site-packages/kat/plot/__pycache__/cold.cpython-35.opt-1.pyc
/usr/lib64/python3.5/site-packages/kat/plot/__pycache__/cold.cpython-35.opt-2.pyc
/usr/lib64/python3.5/site-packages/kat/plot/__pycache__/cold.cpython-35.pyc
/usr/lib64/python3.5/site-packages/kat/plot/__pycache__/colormaps.cpython-35.opt-1.pyc
/usr/lib64/python3.5/site-packages/kat/plot/__pycache__/colormaps.cpython-35.opt-2.pyc
/usr/lib64/python3.5/site-packages/kat/plot/__pycache__/colormaps.cpython-35.pyc
/usr/lib64/python3.5/site-packages/kat/plot/__pycache__/density.cpython-35.opt-1.pyc
/usr/lib64/python3.5/site-packages/kat/plot/__pycache__/density.cpython-35.opt-2.pyc
/usr/lib64/python3.5/site-packages/kat/plot/__pycache__/density.cpython-35.pyc
/usr/lib64/python3.5/site-packages/kat/plot/__pycache__/misc.cpython-35.opt-1.pyc
/usr/lib64/python3.5/site-packages/kat/plot/__pycache__/misc.cpython-35.opt-2.pyc
/usr/lib64/python3.5/site-packages/kat/plot/__pycache__/misc.cpython-35.pyc
/usr/lib64/python3.5/site-packages/kat/plot/__pycache__/profile.cpython-35.opt-1.pyc
/usr/lib64/python3.5/site-packages/kat/plot/__pycache__/profile.cpython-35.opt-2.pyc
/usr/lib64/python3.5/site-packages/kat/plot/__pycache__/profile.cpython-35.pyc
/usr/lib64/python3.5/site-packages/kat/plot/__pycache__/spectra_cn.cpython-35.opt-1.pyc
/usr/lib64/python3.5/site-packages/kat/plot/__pycache__/spectra_cn.cpython-35.opt-2.pyc
/usr/lib64/python3.5/site-packages/kat/plot/__pycache__/spectra_cn.cpython-35.pyc
/usr/lib64/python3.5/site-packages/kat/plot/__pycache__/spectra_hist.cpython-35.opt-1.pyc
/usr/lib64/python3.5/site-packages/kat/plot/__pycache__/spectra_hist.cpython-35.opt-2.pyc
/usr/lib64/python3.5/site-packages/kat/plot/__pycache__/spectra_hist.cpython-35.pyc
/usr/lib64/python3.5/site-packages/kat/plot/__pycache__/spectra_mx.cpython-35.opt-1.pyc
/usr/lib64/python3.5/site-packages/kat/plot/__pycache__/spectra_mx.cpython-35.opt-2.pyc
/usr/lib64/python3.5/site-packages/kat/plot/__pycache__/spectra_mx.cpython-35.pyc
/usr/lib64/python3.5/site-packages/kat/plot/cold.py
/usr/lib64/python3.5/site-packages/kat/plot/colormaps.py
/usr/lib64/python3.5/site-packages/kat/plot/density.py
/usr/lib64/python3.5/site-packages/kat/plot/misc.py
/usr/lib64/python3.5/site-packages/kat/plot/profile.py
/usr/lib64/python3.5/site-packages/kat/plot/spectra_cn.py
/usr/lib64/python3.5/site-packages/kat/plot/spectra_hist.py
/usr/lib64/python3.5/site-packages/kat/plot/spectra_mx.py
/usr/lib64/python3.5/site-packages/kat/spectra.py
/usr/share
/usr/share/doc
/usr/share/doc/kat-2.4.1
/usr/share/doc/kat-2.4.1/AUTHORS.bz2
/usr/share/doc/kat-2.4.1/NEWS.bz2
/usr/share/doc/kat-2.4.1/README.md.bz2
/usr/share/doc/kat-2.4.1/html
/usr/share/doc/kat-2.4.1/html/html
/usr/share/doc/kat-2.4.1/html/html/.buildinfo
/usr/share/doc/kat-2.4.1/html/html/_images
/usr/share/doc/kat-2.4.1/html/html/_images/ccoli_comp.png
/usr/share/doc/kat-2.4.1/html/html/_images/ccoli_gcp.png
/usr/share/doc/kat-2.4.1/html/html/_images/ccoli_hist.png
/usr/share/doc/kat-2.4.1/html/html/_images/contaminant_MP.png
/usr/share/doc/kat-2.4.1/html/html/_images/contaminant_PE.png
/usr/share/doc/kat-2.4.1/html/html/_images/contaminant_all.png
/usr/share/doc/kat-2.4.1/html/html/_images/contaminant_ope1.png
/usr/share/doc/kat-2.4.1/html/html/_images/contaminant_ope2.png
/usr/share/doc/kat-2.4.1/html/html/_images/distanalysis_console.png
/usr/share/doc/kat-2.4.1/html/html/_images/distanalysis_plot.png
/usr/share/doc/kat-2.4.1/html/html/_images/gc_bias_a.png
/usr/share/doc/kat-2.4.1/html/html/_images/gc_bias_b.png
/usr/share/doc/kat-2.4.1/html/html/_images/gc_bias_c.png
/usr/share/doc/kat-2.4.1/html/html/_images/gc_bias_d.png
/usr/share/doc/kat-2.4.1/html/html/_images/heterozygous_perfect_1.png
/usr/share/doc/kat-2.4.1/html/html/_images/heterozygous_perfect_2.png
/usr/share/doc/kat-2.4.1/html/html/_images/heterozygous_real.png
/usr/share/doc/kat-2.4.1/html/html/_images/kat_logo.png
/usr/share/doc/kat-2.4.1/html/html/_images/kmer_spectra1.png
/usr/share/doc/kat-2.4.1/html/html/_images/kmer_spectra_breakdown.png
/usr/share/doc/kat-2.4.1/html/html/_images/pe_v_asm_clean.png
/usr/share/doc/kat-2.4.1/html/html/_images/pe_v_asm_wrong.png
/usr/share/doc/kat-2.4.1/html/html/_images/pe_v_mp_after_density.png
/usr/share/doc/kat-2.4.1/html/html/_images/pe_v_mp_after_shared.png
/usr/share/doc/kat-2.4.1/html/html/_images/pe_v_mp_before_density.png
/usr/share/doc/kat-2.4.1/html/html/_images/pe_v_mp_before_shared.png
/usr/share/doc/kat-2.4.1/html/html/_images/pe_v_pe_1_density.png
/usr/share/doc/kat-2.4.1/html/html/_images/pe_v_pe_1_shared.png
/usr/share/doc/kat-2.4.1/html/html/_images/pe_v_pe_2_density.png
/usr/share/doc/kat-2.4.1/html/html/_images/pe_v_pe_2_shared_1.png
/usr/share/doc/kat-2.4.1/html/html/_images/pe_v_pe_2_shared_2.png
/usr/share/doc/kat-2.4.1/html/html/_images/profile.png
/usr/share/doc/kat-2.4.1/html/html/_images/real_r1_v_r2.png
/usr/share/doc/kat-2.4.1/html/html/_images/real_r1_v_r2_shared.png
/usr/share/doc/kat-2.4.1/html/html/_images/simulated_r1_v_r2.png
/usr/share/doc/kat-2.4.1/html/html/_sources
/usr/share/doc/kat-2.4.1/html/html/_sources/faq.rst.txt
/usr/share/doc/kat-2.4.1/html/html/_sources/index.rst.txt
/usr/share/doc/kat-2.4.1/html/html/_sources/installation.rst.txt
/usr/share/doc/kat-2.4.1/html/html/_sources/kmer.rst.txt
/usr/share/doc/kat-2.4.1/html/html/_sources/using.rst.txt
/usr/share/doc/kat-2.4.1/html/html/_sources/walkthrough.rst.txt
/usr/share/doc/kat-2.4.1/html/html/_static
/usr/share/doc/kat-2.4.1/html/html/_static/ajax-loader.gif
/usr/share/doc/kat-2.4.1/html/html/_static/basic.css
/usr/share/doc/kat-2.4.1/html/html/_static/comment-bright.png
/usr/share/doc/kat-2.4.1/html/html/_static/comment-close.png
/usr/share/doc/kat-2.4.1/html/html/_static/comment.png
/usr/share/doc/kat-2.4.1/html/html/_static/doctools.js
/usr/share/doc/kat-2.4.1/html/html/_static/down-pressed.png
/usr/share/doc/kat-2.4.1/html/html/_static/down.png
/usr/share/doc/kat-2.4.1/html/html/_static/file.png
/usr/share/doc/kat-2.4.1/html/html/_static/jquery-3.2.1.js
/usr/share/doc/kat-2.4.1/html/html/_static/jquery.js
/usr/share/doc/kat-2.4.1/html/html/_static/kat_logo.png
/usr/share/doc/kat-2.4.1/html/html/_static/minus.png
/usr/share/doc/kat-2.4.1/html/html/_static/nature.css
/usr/share/doc/kat-2.4.1/html/html/_static/plus.png
/usr/share/doc/kat-2.4.1/html/html/_static/pygments.css
/usr/share/doc/kat-2.4.1/html/html/_static/searchtools.js
/usr/share/doc/kat-2.4.1/html/html/_static/underscore-1.3.1.js
/usr/share/doc/kat-2.4.1/html/html/_static/underscore.js
/usr/share/doc/kat-2.4.1/html/html/_static/up-pressed.png
/usr/share/doc/kat-2.4.1/html/html/_static/up.png
/usr/share/doc/kat-2.4.1/html/html/_static/websupport.js
/usr/share/doc/kat-2.4.1/html/html/faq.html
/usr/share/doc/kat-2.4.1/html/html/genindex.html
/usr/share/doc/kat-2.4.1/html/html/index.html
/usr/share/doc/kat-2.4.1/html/html/installation.html
/usr/share/doc/kat-2.4.1/html/html/kmer.html
/usr/share/doc/kat-2.4.1/html/html/objects.inv
/usr/share/doc/kat-2.4.1/html/html/search.html
/usr/share/doc/kat-2.4.1/html/html/searchindex.js
/usr/share/doc/kat-2.4.1/html/html/using.html
/usr/share/doc/kat-2.4.1/html/html/walkthrough.html
/usr/share/man
/usr/share/man/man1
/usr/share/man/man1/kat.1.bz2
/usr/share/man/man1/kat_jellyfish.1.bz2
#
mmokrejs commented 6 years ago

It registered with only one python version for some reason. Usually multiple are supported. It should have recognized 3.5 and 3.6 per PYTHON_COMPAT=( python3_{5,6} ) in the *.ebuild files.

mmokrejs commented 6 years ago

Hi @maplesond , please have a look at https://github.com/gentoo/sci/issues/867 , there is a bunch of issues found by @kiwifb . Also check https://github.com/gentoo/sci/tree/master/sci-biology/kat time to time.

kiwifb commented 6 years ago

OK, since I have been called

Running KAT in SECT mode

Input 1 is a sequence file. Counting kmers for input 1 () ... Time taken: 0.1s

KAT SECT completed. Total runtime: 0.1s

Error: Unsupported format fbissey@moonloop /dev/shm/portage/sci-biology/kat-2.4.1/work/KAT-Release-2.4.1/tests $ ../src/kat sect -o temp/sect_length data/sect_length_test.fa data//ecoli.header.jf27 Kmer Analysis Toolkit (KAT) V2.4.1

Running KAT in SECT mode

Loading hashes into memory... done. Time taken: 0.0s

Calculating kmer coverage across sequences ... done. Time taken: 0.0s

Merging matrices ... done. Time taken: 0.1s

KAT SECT completed. Total runtime: 0.2s

but

fbissey@moonloop /dev/shm/portage/sci-biology/kat-2.4.1/work/KAT-Release-2.4.1/tests $ ../src/kat sect -o temp/sect_length /dev/shm/portage/sci-biology/kat-2.4.1/work/KAT-Release-2.4.1/tests/data/sect_length_test.fa /dev/shm/portage/sci-biology/kat-2.4.1/work/KAT-Release-2.4.1/tests/data/ecoli.header.jf27 Kmer Analysis Toolkit (KAT) V2.4.1

Running KAT in SECT mode

Input 1 is a sequence file. Counting kmers for input 1 () ... Time taken: 0.1s

KAT SECT completed. Total runtime: 0.1s

Error: Unsupported format


conclusion we are hitting a limit in number of characters in the argument of `kat sect` and it gets truncated.
maplesond commented 6 years ago

Thanks for the feedback @kiwifb. A couple of questions:

kiwifb commented 6 years ago
kiwifb commented 6 years ago

For the two switches --disable-pykat --enable-pykat-install will fail during make if no python or a wrong version of python is found in the PATH. It will install pykat if a suitable version of python is in the PATH.

But it doesn't make sense. If you --disable-pykat you shouldn't care about installing it or not. pykat-install should either be disregarded or the above combination be treated as an error.

maplesond commented 6 years ago

Hi @kiwifb, 

✔ ~/dev/KAT/tests [develop|✔] 
10:44 $ ./test_sect.sh 
Kmer Analysis Toolkit (KAT) V2.4.1

Running KAT in SECT mode
------------------------

Loading hashes into memory... done.  Time taken: 0.0s

Calculating kmer coverage across sequences ... done.  Time taken: 0.0s

Merging matrices ... done.  Time taken: 0.2s

KAT SECT completed.
Total runtime: 0.2s

Kmer Analysis Toolkit (KAT) V2.4.1

Running KAT in SECT mode
------------------------

Loading hashes into memory... done.  Time taken: 0.0s

Calculating kmer coverage across sequences ... done.  Time taken: 0.0s

Merging matrices ... done.  Time taken: 0.2s

KAT SECT completed.
Total runtime: 0.2s

The error you are seeing is raised by jellyfish when trying to interpret the format of the hash file. If the file doesn't exist it would also raise the same error you are seeing, so a truncation of the command line is a plausible explanation. On my system it looks like I have can have command lines up to 2MB in length:

✔ ~/dev/KAT [develop|✔] 
10:40 $ getconf ARG_MAX
2097152

Maybe this is less on your system? But I find it hard to believe it would be any smaller than 1KB.

kiwifb commented 6 years ago

Hum...

fbissey@moonloop ~/sandbox/KAT-Release-2.4.1 $ getconf ARG_MAX
2097152

so not that. I'll have a think about debugging this some more.

For the various linking issues you can inspire yourself from https://github.com/kiwifb/sci/blob/kat/sci-biology/kat/files/kat-2.4.1-ignore-bundled-deps.patch with the caveat that you want to use your own boost at this stage.

maplesond commented 6 years ago

I've had a go at address the underlinking issue. However, I suspect that this will break unit testing on Mac platform in the process as I had to use rpaths to make the boost_timer library available at runtime for the unit tests. I'm not sure why it was necessary to do this as the main kat executable build fine without requiring dynamic links to any boost libs. 94c14d057067f14008b675713d9de36e7a0186a0

I should also add that while I suspect the kat library won't under link for you as you have boost installed to the system, it still does for me, albeit far less badly, as I don't have boost installed system-wide.

maplesond commented 6 years ago

With regards to the doc changes. Wouldn't the change you suggest implies that man pages will automatically get built with html, even if the user requests to only build html. This isn't really ideal and is confusing for the user. Is there another way around the issue you can think of?

kiwifb commented 6 years ago

I guess for the doc change we may be able to do something else if you want fine grained. But at the moment we have 

# Hook in conventional make targets for building docs
all: html man

so by default when you run make at the top level both man and html will want to be built. They cannot be built in parallel.

kiwifb commented 6 years ago

Of course, now I cannot reproduce the failure. Parallel build can be so volatile in their failures. Fortunately I have old logs. The problem is a race condition in the creation of the doctree subfolder. Both sphinx processes will want to create it if it is missing. And one of processes finds it tries to create a folder that is already there. So timing is very important in the reproduction of the problem.

On the other hand just creating the folder before any call to sphinx-build may be enough to prevent this particular problem.

kiwifb commented 6 years ago

Regarding the linking. I think I know the sources of your problems. The libkat library is dynamically linked to boost so it may very well still require access to it. I'll note that using (DY)LD_LIBRARY_PATH for testing with autotools is very frequent. But you could also try to link boost statically to libkat by doing it the same way in libkat than you do in the kat executable.

kiwifb commented 6 years ago

Also linking order can have some impact on stuff, especially so with some linking flags. So I would move libkat to the top, because it uses boost libraries, and the linker may want to know about the symbols in libkat that need boost before bringing in the boost libraries.

maplesond commented 6 years ago

With regards to documentation. One simple and clean solution is just to create and install the man pages by default. That would avoid the race condition, and be suitable for most users. There is an online html version of the docs that get automatically build with every release on readthedocs which is referenced in the README.md file. I'll go ahead and make that change. 239f5488a2241249b21148b49071bda38681434a

maplesond commented 6 years ago

I've added a fix for the two pykat options. Now if pykat is disabled, so will pykat install. 38c9b37abf2def6917979bf2ec4f70f54011e5e2

maplesond commented 6 years ago

Thanks @kiwifb and @mmokrejs for all your help and patience with this. Assuming the change above are suitable, I think there are three remaining issues:

kiwifb commented 6 years ago

Ok, was a public holiday yesterday so I spent 0 time on this then. So now I am taking a look at your develop branch.

kiwifb commented 6 years ago

I see that KAT_SITE_PKGS is defined unilaterally in some Makefile.am, this doesn't appear to be documented but AX_PYTHON_DEVEL runs a AC_SUBST command with PYTHON_SITE_PKG which would be a better fit to start with. Of course that doesn't solve your problem for custom install but this is a better base. Although I see that you do use PYTHON_INTERP_SITE_PKGS which again is only defined in config.h - which is not shipped.

maplesond commented 6 years ago

Hi @kiwifb , no problem. Apologies in advance for the essay but I think it might be helpful to clarify how python is used by KAT and add a little bit of history about the project so that it makes sense.

So initially, KAT was solely a C++ project, with some plotting using gnuplot. However, since then we've replaced the gnuplot plotting with python matplotlib scripts and we analyse some of the distribution data that KAT produces. Ideally, this would all be C++ code too but life's too short. So to workaround this, we wanted to be able to use the python code in two ways. First of all as standalone scripts (easy), and second via python embedding within the host C++ process (a little trickier). When it works this gives the user the best of both worlds, however it does cause some configuration headaches.

The other thing to keep in mind about KAT is that it's intended mainly for HPC environments, or at least for machines with a minimum of 32GB RAM, ideally 256GB+, (depends on the datasets being used). So we haven't focused very heavily on support of Mac or desktop use. (However, we are looking at using different K-mer counting methods in the future which should reduce this mem requirement.) As a result, we almost always envisage KAT being installed to custom locations i.e. with ./configure --prefix=<some_dir>. This is one of the reasons why the python installation to a system location failed (i.e. lack of testing for this scenario on my side).

So I've added a few more changes into the develop branch. Basically, for KAT to detect the python scripts for embedding I now put a copy in $(datadir)/kat/scripts/kat. This makes it far more predictable to find, and simplifies some code in lib/include/kat/kat_fs.hpp. I've also changed the location the python packages get installed (for users that wish to import the libraries in python or run the scripts directly). When a prefix is given to configure, this still gets installed to $(prefix)/lib/python<version>/site-packages. This is important for our use cases where we are maintaining multiple versions at the same time. Maybe we could use virtual environments for this instead, but this seems like an ok solution. The only downside is that the user needs to update their PYTHONPATH at runtime. When no prefix is passed to configure, then instead of installing to /usr/local/lib/python<version>/site-packages, we lookup the site packages used by the python interpreter that is found on the path and install to their instead, this may or may not be in the system location, and is probably what the users are expecting to happen.

BOOST: I agree regarding AX_BOOST. KAT used to use this but that didn't work so great. Neither did including boost git submodules, which is why I opted for boost's own bcp method and directly embedded the subset in KAT.

KAT SECT test: I haven't actually tried expanding out into a long path yet... I'll try that later today.

kiwifb commented 6 years ago

Just to be clear my job title is "Supercomputing Services and Support Consultant" so I am hearing your use case loud and clear. I would expect a judicious use of module/lmod (or whatever other system you have) to go a long way.

I think I need to understand what you are doing better. My personal opinion on this, is that a build system for the desktop using autotools and other friends can be easily be put in its own sandbox. Playing along with compartmented python modules is slightly more tricky but not too much.

mmokrejs commented 6 years ago

Hi @maplesond , most of the issues I am not really up to to comment but regarding the python's site-packages paths ...

We have hooks to run setup.py for every python version installed in Gentoo. For example, that is, three times if I have python-3.{4,5,6} installed. The setup.py is called with --prefix argument and I am not sure here but maybe a destdir variable is passed down too?, so that the install procedure can install the files not straight into $prefix but into "$tmppath"/"$prefix". Otherwise SANDBOX_VIOLATION is triggered. If the destdir variable is needed to be defined anywhere in setup.py or not ... maybe it is faster to look into some python-based packages to see if they define it somewhere? I never did that myself so I can't say. Say biopython's integration is working well, whatever they employ in setup.py.

Provided you set the prefix only once during the configure exec, we are then unable to register your python modules into more than just a single python version.

maplesond commented 6 years ago

Hi @mmokrejs, hmm. If it is being installed to "$tmppath"/"$prefix"/lib/python<version>/site-packages then something has gone wrong. Where would the $tmppath part come from?

To elaborate on the new setup (latest develop commit), there are 3 options which I deal with for installing python scripts:

Is there any chance the $(prefix) part could get itself prefixed with $tmppath, to explain the issues you are seeing?

mmokrejs commented 6 years ago

If it is being installed to "$tmppath"/"$prefix"/lib/python/site-packages then something has gone wrong. Where would the $tmppath part come from?

The package manager moves the contents of "$tmppath"/"$prefix" to "$prefix" later with root privileges. That is the required procedure. Yes, maybe --prefix="$DESTDIR"/"$PREFIX" is the way how it works? I am just not knowledgeable enough to confirm. I suspect that maybe setup.py --prefix=$prefix --destdir=$destdir is anticipated.

At least in case of Makefile's there are both, prefix and DESTDIR variables needed. Don't know about setup.py common variables. They are comon for sure, almost every python package is automatically picked up and just works (means gets registered with every python version we have installed system-wide). Just stick to the usual setup.py found in other packages.

maplesond commented 6 years ago

@kiwifb , with regards to to sect long paths don't seem to affect the behaviour on my system:

make[2]: Entering directory '/tmp/kat_really_long_path_to_test_if_sect_has_some_issues_with_this_jdlsjfkdlsjfkldsjfkdslfjdsklfjdsklfjdsklfjdskl/parpyparpppapapshjkgfhjkhgfdsjkfhjkdsf/package/KAT-Release-2.4.1/tests'
  CXX      check_jellyfish.o
  CXX      check_spectra_helper.o
  CXX      check_compcounters.o
  CXX      check_main.o
  CXX      gtest/src/libgtest_la-gtest-all.lo
  CXX      gtest/src/libgtest_main_la-gtest_main.lo
  CXXLD    libgtest.la
ar: `u' modifier ignored since `D' is the default (see `U')
  CXXLD    libgtest_main.la
  CXXLD    check_unit_tests
ar: `u' modifier ignored since `D' is the default (see `U')
make[2]: Leaving directory '/tmp/kat_really_long_path_to_test_if_sect_has_some_issues_with_this_jdlsjfkdlsjfkldsjfkdslfjdsklfjdsklfjdsklfjdskl/parpyparpppapapshjkgfhjkhgfdsjkfhjkdsf/package/KAT-Release-2.4.1/tests'
make  check-TESTS
make[2]: Entering directory '/tmp/kat_really_long_path_to_test_if_sect_has_some_issues_with_this_jdlsjfkdlsjfkldsjfkdslfjdsklfjdsklfjdsklfjdskl/parpyparpppapapshjkgfhjkhgfdsjkfhjkdsf/package/KAT-Release-2.4.1/tests'
make[3]: Entering directory '/tmp/kat_really_long_path_to_test_if_sect_has_some_issues_with_this_jdlsjfkdlsjfkldsjfkdslfjdsklfjdsklfjdsklfjdskl/parpyparpppapapshjkgfhjkhgfdsjkfhjkdsf/package/KAT-Release-2.4.1/tests'
PASS: test_sect.sh
PASS: check_unit_tests
PASS: test_hist.sh
PASS: test_gcp.sh
PASS: test_comp.sh
============================================================================
Testsuite summary for kat 2.4.1
============================================================================
# TOTAL: 5
# PASS:  5
# SKIP:  0
# XFAIL: 0
# FAIL:  0
# XPASS: 0
# ERROR: 0
============================================================================

Running ./test_sect.sh in isolation is fine too.

kiwifb commented 6 years ago

That's really an odd problem. I may have to test on another system.

maplesond commented 6 years ago

Hi @mmokrejs, Just so I understand are you doing something like this: ./configure --prefix=/usr, make, make install DESTDIR=/tmp

If so, I haven't tried or considered this scenario before. I can see now that this would break the python prefix location.

I've updated the develop branch to address this: 30d2e798b32f701daf4b0d93ea6b128cbccec58f

mmokrejs commented 6 years ago

./configure --prefix=/usr, make, make install DESTDIR=/tmp

This is what generally happens form packages which are purely based on configure & make approach.

For purely pythonic packages the setup.py is called with arguments ... with which I am not certain (as I explained above). Would you move the pythonic stuff into a separate package, things would be clearer to us and to the build system.

I hope @kiwifb or @mgorny could shed more light on this.

https://wiki.gentoo.org/wiki/Project:Python/distutils-r1 https://devmanual.gentoo.org/eclass-reference/distutils-r1.eclass/index.html

mgorny commented 6 years ago

First of all, I'm sorry but I won't be able to read all the comments above. Therefore, I'm not really sure what I'm supposed to say here.

If it's about running setup.py, then its equivalent of Makefile DESTDIR option is --root= parameter to install, e.g. you'd call something like:

$(PYTHON) setup.py install --prefix=... --root=$(DESTDIR)

However, note that I haven't tested the behavior with empty DESTDIR, so it may need extra conditions.

As for installing Python modules 'manually', you really shouldn't reinvent the rules. Automake has some built-in stuff for Python that handles byte-compiling etc. Look at info automake, search for Python and it has quite detailed instructions on how to use it. It also takes care of determining the correct directories.

mmokrejs commented 6 years ago

$(PYTHON) setup.py install --prefix=... --root=$(DESTDIR)

@mgorny thank you, this is exactly what we were after. Even more thanks for the other comments.

maplesond commented 6 years ago

Thanks @mgorny. @mmokrejs, I've made these suggested changes to the develop branch. 580f033cf5001c7857e792e0b49a646b5eb4858c

mgorny commented 6 years ago

FTR, you're still wrongly assuming /usr/lib/python*. If this package supports PyPy, then the path will be even more arbitrary.

Generally the best thing would be to split the Python part into separate package, and possibly upload it to pypi so that it could be installed independently for multiple versions of Python.

maplesond commented 6 years ago

Hi @mgorny, I'm confused, are you checking the master branch, or develop? In the master branch, there was a bug whereby if the configure prefix wasn't specified it was incorrectly assuming /usr/lib/python3 (python2 is disallowed). This should all be rectified in the develop branch. I'd like to get the ok from you guys first before merging into master and cutting a new release. AFAIK I use which ever python is picked up first on the path. On my system for example it uses ~/software/anaconda3/bin/python3. I can pick up all the associated directories, e.g. site-packages from this using a bit of python and distutils, which is done in the m4 macros. This seems to work as expected on my system (with or without destdir or prefix).

Ideally, I'd like to avoid splitting out into a separate project. Given the way I'm using python here it's just going to be difficult from a maintenance perspective. If you can provide an example where the current solution doesn't work then I'll reconsider though.

mgorny commented 6 years ago

I'm talking at the commit you linked. You're using /usr/lib while /usr/lib64 is actually used by our Python on amd64. However, you can't really hardcode any path there, you need to grab it from Python itself. I'm sorry but I don't have the command handy. You can look how autoconf/automake gets it.