core dump processing real time feed

[cchayden]

I have been using libais 0.16 to process a real-time feed from ORBCOMM. It has been running without a single error for a year. Today, for the first time, we saw a core dump with the following message: " what(): bitset::set: __position (which is 32) >= _Nb (which is 32)"

Has anyone else seen this?

Added: The message that is causing the failure is: !AIVDM,1,1,,B,677IKl=HQGw004D0@02000000000,0*5B

This appears to be a type 6 message.

[cchayden]

I have investigated further, and I have identified the cause of the core dump. In ais6.cpp, on line 257, the line: spare = bs.ToUnsignedInt(119, 49); is collecting the rest of the spare bits of the message. But ToUnsignedInt only has room for 32 bits, so as soon as it passes that mark, bs_tmp.set on line 156 of ais.cpp fails, causing a core dump.

I see in some of the other parsing functions in ais6, when there is need to read more than 32 bits of spare, that the code uses two spares. This needs to be done for ais6_1_5 as well.

[enriquetuya]

@schwehr As you have mention in https://github.com/GlobalFishingWatch/GFW-Tasks/issues/591, the error we are experiencing is the same as @cchayden mentions. The exact same message is giving us problems. We have had issues with 2017-11-04 data and also 2017-11-06. I will research specific messages on 06 and add it to this ticket.

[schwehr]

Taking a look. I am able to reproduce trouble with the sample line.

[----------] 1 test from Ais6 [ RUN ] Ais6.Issue162 F1115 22:28:03.883800 911381 logging.cc:2239] assert.h assertion failed at tlibais/ais.cpp:148 in unsigned int libais::AisBitset::ToUnsignedInt(const size_t, const size_t) const: len <= 32 Check failure stack trace: @ 0x7f87bc4f4cca __assert_fail @ 0x7f87bfb6be6b libais::AisBitset::ToUnsignedInt() @ 0x7f87bfb77c40 libais::Ais6_1_5::Ais6_1_5() @ 0x454185 absl::MakeUnique<>() @ 0x452496 libais::(anonymous namespace)::Ais6_Issue162_Test::TestBody()

[cchayden]

I did submit a PR that fixes this problem 10 days ago.

[schwehr]

https://github.com/schwehr/libais/commit/65cb1b12d65ebab97a34f4e54daaae81eab4b949 Splits spare into spare and spare2. Then @cchayden has the rest of the basic required code by initializing spare2 correctly. Then I'll follow up with a simple test.

More 6:1:5 with some more diversity would be helpful for test coverage.

[enriquetuya]

@schwehr here you have a few more samples that causes the error:

AIVDM,1,1,,B,677IKl=HQGw004D0@02000000000,0*5B
AIVDM,1,1,,B,69NSH@AI746004D0@06B00000000,0*46
AIVDM,1,1,,A,676ur660AE4F04D0@0;P00000000,0*3B
AIVDM,1,1,,A,69NSKE20ACVL04D0@03P00000000,0*43

@cchayden next time you create a PR try to add a Connects to #ISSUE on the description so it is displayed on the github issue history.

[enriquetuya]

@schwehr latest version produces a segmentation fault error. To reproduce:

libais_162_issue.sh

#!/bin/bash

mkdir libais_test
cd libais_test

virtualenv venv
source venv/bin/activate
pip --cache-dir ./pip-cache install \
    git+https://github.com/schwehr/libais@master
python -c 'import ais; print ais.decode("801tME0j60E04QJp1hhL10G20DH1",0)'

[schwehr]

The segfault is from 8:200:24. EU RIS. Working on it now.

[schwehr]

I think I've fixed the two separate issues. I'm closing this issue now. Reopen if that doesn't completely address both. If you hit an issue with a different message, please open a new issue.