pycbf 0.9.7 does not pass tests out of the box.

[picca]

Hello, I am preparing the cbflib 0.9.7 package for Debian.

I now have a package built with only the Python3 version pf pycbf. I want to add autopkgtest in order to have integration tests which checks that nothing break when a dependency change. So my first attempt was to execute the 4 test of the pycbf directory. But it end-up with this error

$python3 pycbf_test1.py 
Traceback (most recent call last):
  File "/home/experiences/instrumentation/picca/debian/science-team/cbflib/pycbf/pycbf_test1.py", line 4, in <module>
    object.read_file(b"../img2cif_packed.cif",pycbf.MSG_DIGEST)
  File "/usr/lib/python3/dist-packages/pycbf.py", line 2009, in read_file
    return _pycbf.cbf_handle_struct_read_file(self, filename, headers)
TypeError: in method 'cbf_handle_struct_read_file', argument 2 of type 'char *

This issue cause also troubles when I try to run the dxtbx test suite.

This problem seems to be related to the handling of string/bytes.

Can you reproduce this issue on your system ?

thanks

[picca]

I saw that some binary files are used to do some test. Could you explain what is the right way to test thaht cbflib is working.

thanks

[picca]

I did a mistake during the build of cbflib. I do not have nuweb so I generated the setup.py for Python3 from the m4/setup.py whcih does not contain the SWIG_PYTHON_STRICT_BYTE_CHAR flag., with this fag it is better. nevertheless It would be great if you could give me the recipes to run all the test.

thanks

[yayahjb]

I last ran all tests under Debian 11 using make tests. Please point me to a download url for your kit.

On Tue, Aug 9, 2022, 5:49 AM picca @.***> wrote:

I saw that some binary files are used to do some test. Could you explain what is the right way to test thaht cbflib is working.

thanks

— Reply to this email directly, view it on GitHub https://github.com/yayahjb/cbflib/issues/55#issuecomment-1209161098, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABB6EAOUZGAVVDLXGJYPKT3VYISSNANCNFSM557VGFJQ . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[picca]

The Debian package of cbflib is here

https://salsa.debian.org/science-team/cbflib

Do you know how to build a Debian package ?

apt install git-buildpackage
gbp clone https://salsa.debian.org/science-team/cbflib
cd cbflib && gbp buildpackage

Cheers, I will be in hollydays until 29 of August.

[ndevenish]

This Dockerfile runs everything under make tests:

FROM debian:11

RUN apt-get update && \
    apt-get install -y \
            build-essential m4 texlive wget git rsync cmake gfortran links \
            python2 python-setuptools python2-dev \
            python3 python3-setuptools python3-dev python3-matplotlib python3-numpy

# Create a user to build; the CBFlib tarballs encode user ID, so we cannot
# build as root without rewriting the tar calls to "tar --no-same-owner".
RUN useradd -m cbfbuilder
USER cbfbuilder

# Get CBFlib
# NOTE: We want 0.9.7 but the actual tagged commit has various build
#       errors e.g. nuweb at least. So get the latest, even though this
#       strictly isn't testing the actual release
RUN git clone https://github.com/yayahjb/cbflib /home/cbfbuilder/cbflib
WORKDIR /home/cbfbuilder/cbflib

# Download the test data files explicitly
RUN make $(pwd)/../CBFlib_0.9.7_Data_Files_Input $(pwd)/../CBFlib_0.9.7_Data_Files_Output

# Build NuWeb explicitly (rather than implicitly when running tests)
RUN make $(pwd)/bin/nuweb

RUN make all

# Build other things that "all" doesn't include, but tests do... like pycbf
RUN make $(pwd)/pycbf/_pycbf.so $(pwd)/py2cbf/_py2cbf.so

RUN make tests

Albeit with the main branch, not the tagged release, because of various failures (e.g. the built-in nuweb compilation).

It passes... most tests. The py2cbf tests don't work properly "Youneedtogetnumpyandmatplotlibtoseethedata" which I can't fix because debian 11 as far as I can tell appears not to include matplotlib and numpy (which makes sense, as they would be ancient).

It's hard to tell definitively otherwise because it appears for the most part that running the tests just runs them, it doesn't actually check they pass; test failures don't actually cause a failed exit code (e.g. the only way to tell the above py2cbf error failed is to grep for Error 1 (ignored)). However, there does appear to be a single py3cbftests error:

(cd /home/cbfbuilder/cbflib/pycbf; grep -v "__builtins__" /home/cbfbuilder/cbflib/pycbf_test1_orig.out | \
    ▏   ▏ grep -v "__add__" | grep -v "Foundthebinary" > pycbf_test1_orig.out; \
    ▏   ▏ grep -v "__builtins__"  pycbf_test1.out | \
    ▏   ▏ grep -v "__add__" | grep -v "Foundthebinary" |diff -u -b - pycbf_test1_orig.out)
--- -   2022-08-19 20:24:05.555015966 +0000
+++ pycbf_test1_orig.out    2022-08-19 20:24:05.552782261 +0000
@@ -142,7 +142,7 @@
 col:array_idtype:word
 Val:1 11
 col:binary_idtype:bnry
-7.8804e+07
+2.116e+07
 ( 5.29e+06 ,  )
 [ 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 ]  56 0
 [ 409 , 410 , 388 , 402 , 407 , 408 , 412 , 403 , 427 , 416 , 420 , 396 , 393 , 426 , 463 , 457 , 440 , 422
make: [Makefile:2700: py3cbftests] Error 1 (ignored)

So it does indeed seem to be true that the tests do not pass out-of-the-box, as long as that box is the latest main not the release, and are run on a clean debian 11.