search

kbandla / dpkt

fast, simple packet creation / parsing, with definitions for the basic TCP/IP protocols
Other
1.09k stars 270 forks source link

fix endianness issues in unit tests #615

Closed obormot closed 2 years ago

coveralls commented 2 years ago

Coverage Status

Coverage increased (+0.0001%) to 99.815% when pulling 9547a5134d69d3e902f121ea1545d45480823d9e on issue#505 into 04bd8ec675830bedf5a4e7fe52873f2da65e3bbb on master.

obormot commented 2 years ago

All pcapng unit tests now PASS on both big endian and little endian platforms. The 2 loopback.py tests still fail. 

# uname -m
s390x

# tox -e py38
GLOB sdist-make: /opt/dpkt/setup.py
py38 inst-nodeps: /opt/dpkt/.tox/.tmp/package/1/dpkt-1.9.7.2.zip
py38 installed: attrs==21.2.0,coverage==6.0.1,dpkt==1.9.7.2,iniconfig==1.1.1,packaging==21.0,pluggy==1.0.0,py==1.10.0,pyparsing==2.4.7,pytest==6.2.5,pytest-cov==3.0.0,toml==0.10.2
py38 run-test-pre: PYTHONHASHSEED='1413270695'
py38 run-test: commands[0] | py.test --cov=dpkt dpkt
=========================================================================================== test session starts ============================================================================================
platform linux -- Python 3.8.10, pytest-6.2.5, py-1.10.0, pluggy-1.0.0 -- /opt/dpkt/.tox/py38/bin/python
cachedir: .tox/py38/.pytest_cache
rootdir: /opt/dpkt, configfile: setup.cfg
plugins: cov-3.0.0
collected 425 items
...
dpkt/loopback.py::test_ethernet_unpack FAILED                                                                                                                                                        [ 54%]
dpkt/loopback.py::test_ip_unpack PASSED                                                                                                                                                              [ 54%]
dpkt/loopback.py::test_ip6_unpack FAILED                                                                                                                                                             [ 54%]
...
dpkt/pcapng.py::test_shb PASSED                                                                                                                                                                      [ 61%]
dpkt/pcapng.py::test_idb PASSED                                                                                                                                                                      [ 61%]
dpkt/pcapng.py::test_epb PASSED                                                                                                                                                                      [ 62%]
dpkt/pcapng.py::test_simple_write_read PASSED                                                                                                                                                        [ 62%]
dpkt/pcapng.py::test_pcapng_header PASSED                                                                                                                                                            [ 62%]
dpkt/pcapng.py::test_filter PASSED                                                                                                                                                                   [ 62%]
dpkt/pcapng.py::test_dispatch PASSED                                                                                                                                                                 [ 63%]
dpkt/pcapng.py::test_loop PASSED                                                                                                                                                                     [ 63%]
dpkt/pcapng.py::test_writer_validate_instance PASSED                                                                                                                                                 [ 63%]
dpkt/pcapng.py::test_writepkt_epb_ts PASSED                                                                                                                                                          [ 63%]
dpkt/pcapng.py::test_writer_validate_le PASSED                                                                                                                                                       [ 64%]
dpkt/pcapng.py::test_writer_validate_be PASSED                                                                                                                                                       [ 64%]
dpkt/pcapng.py::test_writepkt_no_time PASSED                                                                                                                                                         [ 64%]
dpkt/pcapng.py::test_writepkt_snaplen PASSED                                                                                                                                                         [ 64%]
dpkt/pcapng.py::test_writepkt_with_time PASSED                                                                                                                                                       [ 64%]
dpkt/pcapng.py::test_writepkts PASSED                                                                                                                                                                [ 65%]
dpkt/pcapng.py::test_shb_header PASSED                                                                                                                                                               [ 65%]
dpkt/pcapng.py::test_shb_bom PASSED                                                                                                                                                                  [ 65%]
dpkt/pcapng.py::test_shb_version PASSED                                                                                                                                                              [ 65%]
dpkt/pcapng.py::test_no_idb PASSED                                                                                                                                                                   [ 66%]
dpkt/pcapng.py::test_idb_opt_offset PASSED                                                                                                                                                           [ 66%]
dpkt/pcapng.py::test_idb_linktype PASSED                                                                                                                                                             [ 66%]
dpkt/pcapng.py::test_readpkts PASSED                                                                                                                                                                 [ 66%]
dpkt/pcapng.py::test_next PASSED                                                                                                                                                                     [ 67%]
dpkt/pcapng.py::test_repr PASSED                                                                                                                                                                     [ 67%]
dpkt/pcapng.py::test_idb_opt_err PASSED                                                                                                                                                              [ 67%]
dpkt/pcapng.py::test_custom_read_write PASSED                                                                                                                                                        [ 67%]
dpkt/pcapng.py::test_multi_idb_writer PASSED                                                                                                                                                         [ 68%]
dpkt/pcapng.py::test_pcapng_block_pack PASSED                                                                                                                                                        [ 68%]
dpkt/pcapng.py::test_pcapng_block_unpack PASSED                                                                                                                                                      [ 68%]
dpkt/pcapng.py::test_epb_unpack PASSED                                                                                                                                                               [ 68%]
dpkt/pcapng.py::test_epb_unpack_length_mismatch PASSED                                                                                                                                               [ 68%]
dpkt/pcapng.py::test_pcapng_block_len_no_opts PASSED                                                                                                                                                 [ 69%]
dpkt/pcapng.py::test_reader_file_descriptor PASSED                                                                                                                                                   [ 69%]
dpkt/pcapng.py::test_posttest PASSED                                                                                                                                                                 [ 69%]
...
TOTAL                      10251     33    99%

========================================================================================= short test summary info ==========================================================================================
FAILED dpkt/loopback.py::test_ethernet_unpack - assert 131074 == 33554944
FAILED dpkt/loopback.py::test_ip6_unpack - assert 402653184 == 24
========================================================================== 2 failed, 423 passed, 4 warnings in 150.12s (0:02:30) ===========================================================================
obormot commented 2 years ago

All unit tests now pass

# uname -m
s390x

# tox -e py38
...
=============================================================================== 425 passed, 4 warnings in 134.03s (0:02:14) ================================================================================
_________________________________________________________________________________________________ summary __________________________________________________________________________________________________
  py38: commands succeeded
  congratulations :)
mcepl commented 2 years ago

Hmm, I have apparently missed something yet. With this patch on 1.9.7.2 tarball, I get:

[   77s] ____________________________ test_custom_read_write ____________________________
[   77s] 
[   77s]     def test_custom_read_write():
[   77s]         """Test a full pcapng file with 1 ICMP packet"""
[   77s]         buf = define_testdata().valid_pcapng
[   77s]         fobj = BytesIO(buf)
[   77s]     
[   77s]         # test reading
[   77s]         reader = Reader(fobj)
[   77s]         assert reader.snaplen == 0x40000
[   77s]         assert reader.datalink() == DLT_EN10MB
[   77s]     
[   77s]         assert reader.idb.opts[0].data.decode('utf-8') == '\\Device\\NPF_{3BBF21A7-91AE-4DDB-AB2C-C782999C22D5}'
[   77s]         assert reader.idb.opts[2].data.decode('utf-8') == '64-bit Windows 8.1, build 9600'
[   77s]     
[   77s]         ts, buf1 = next(iter(reader))
[   77s]         assert ts == 1442984653.2108380
[   77s]         assert len(buf1) == 74
[   77s]     
[   77s]         assert buf1.startswith(b'\x08\x00\x27\x96')
[   77s]         assert buf1.endswith(b'FGHI')
[   77s]         fobj.close()
[   77s]     
[   77s]         # test pcapng customized writing
[   77s]         shb, idb, epb = define_testdata().shb_idb_epb
[   77s]     
[   77s]         fobj = BytesIO()
[   77s]         writer = Writer(fobj, shb=shb, idb=idb)
[   77s]         writer.writepkt(epb, ts=1442984653.210838)
[   77s]         # .valid_pcapng buf was collected on a little endian system
[   77s] >       assert fobj.getvalue() == buf
[   77s] E       assert b"\n\r\r\n\x0...0\x00\x00\x84" == b"\n\r\r\n|\x...4\x00\x00\x00"
[   77s] E         At index 4 diff: b'\x00' != b'|'
[   77s] E         Full diff:
[   77s] E           (
[   77s] E         -  b'\n\r\r\n|\x00\x00\x00M<+\x1a\x01\x00\x00\x00\xff\xff\xff\xff\xff\xff\xff\xff'
[   77s] E         ?            -            ^^^        ----
[   77s] E         +  b'\n\r\r\n\x00\x00\x00|\x1a+<M\x00\x01\x00\x00\xff\xff\xff\xff\xff\xff\xff\xff'
[   77s] E         ?                        ^    +++++++...
[   77s] E         
[   77s] E         ...Full output truncated (36 lines hidden), use '-vv' to show
[   77s]

Fortunately, it is just the only test which fails now on s390x. Full build log illustrates all packages used and steps taken.

What I am missing?

obormot commented 2 years ago

Hmm, I have apparently missed something yet. With this patch on 1.9.7.2 tarball, I get:

[....]
[   77s]         # .valid_pcapng buf was collected on a little endian system
[   77s] >       assert fobj.getvalue() == buf
[   77s] E       assert b"\n\r\r\n\x0...0\x00\x00\x84" == b"\n\r\r\n|\x...4\x00\x00\x00"
[....]

Fortunately, it is just the only test which fails now on s390x. Full build log illustrates all packages used and steps taken.

What I am missing?

Looks like the diff wasn't fully applied? The assert above (assert fobj.getvalue() == buf) shouldn't have been run on the big endian system. I've added an if condition there but it's missing in the pytest output above

This: https://github.com/kbandla/dpkt/pull/615/files#diff-b7ac13862bd7e7717d0e8da77c39e82c27cd64af621798741ec74ae3df39c9a6R1151

mcepl commented 2 years ago

This: https://github.com/kbandla/dpkt/pull/615/files#diff-b7ac13862bd7e7717d0e8da77c39e82c27cd64af621798741ec74ae3df39c9a6R1151

You are right. Thank you, that helped.