something wrong with the message type

[cchayden]

Has anyone else seen the following problem? It occurs intermittently,

I receive and parse a message. I get the message id from message['id'] I test: if message_id in (1, 2, 3, 18, 19, 27): do something

On the if statement, python throws an exception: list assignment index out of range (IndexError)

It is accompanied by an error: RuntimeWarning: tp_compare didn't return -1 or -2 for exception

This is confusing because the if statement is not doing an index assignment. My guess is that internally python is comparing message_id with the values in the tuple, and one of them does not compare properly. Perhaps because message_id came from C++ and was not properly converted to a legal python value. When I try to print it, message_id prints as "6".

I started seeing this on Nov 13. On Nov 15, there were six such errors around 00:42 UTC and another six at 12:04.

If it would help, I can get the raw messages that triggered these errors.

[schwehr]

Raw messages triggering trouble would definitely help.

[pwoods25443]

I have the same problem when testing message['id'] and I believe that I have isolated the bad message. In my case the bad message was parsing just fine and the error occurred on the following message. I have isolated it down to this, working from master:

import ais

# works
print 1 in (1,2,3)

# works
print ais.decode("63aIjB@0UBf`04@0@0000:00000000000000000", 2)

# fails after the print
print 1 in (1,2,3)

Output

True
{u'cap_reserved': [0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L], u'retransmit': True, u'seq': 0L, u'dac': 1L, u'mmsi': 244740681L, u'repeat_indicator': 0L, u'capabilities': [0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 1L, 1L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L], u'mmsi_dest': 2444010L, u'spare': 0L, u'ack_dac': 1L, u'fi': 4L, u'spare2': 0L, u'id': 6L}
True
Exception IndexError: 'list assignment index out of range' in <module 'threading' from '/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.pyc'> ignored

It appears that something happens in decode() that breaks python

[schwehr]

I'm going to write some tests for 6:1:4 messages.

// 6 1 4 205323000 !AIVDM,1,1,,A,633krv00OEGl04@0Hb00020000000000000000000000000000000000000,2*30
// 6 1 4 205317690 !AIVDM,1,1,,A,633kV>P0OEH@04@0@2000200000000000000000000000000T7L08611020,2*54
// 6 1 4 244850179 !AIVDM,1,1,,B,63aPN0h0UBfj04@0@2000200000000000000000000000000T7L08611020,2*09
// 6 1 4 235095657 !AIVDM,1,1,,B,63P=6J@0WGQt04@0Hb00020000000000000000000000000000000000000,2*1A
// 6 1 4 244110350 !AIVDM,1,1,,A,63`kD3`0UBf`04@0Hb00020000000000000000000000000000000000000,2*70
// '63aIjB@0UBf`04@0@0000:00000000000000000', 2

Note the non-zero data in some of the spare blocks :(

[jamtho]

In function https://github.com/schwehr/libais/blob/538908f2e0cf9067553a7b9f9e158dbe98a47e3a/src/libais/ais_py.cpp#L409-L436

Two pylists of length 26 are constructed. However the for loop runs to index 128/2, and calls PyList_SetItem() with this index for both lists. This presumably writes beyond the end of a malloc'd block.

Changing the calls to PyList_New(128/2) removes the IndexError on my system.

[schwehr]

Thanks @jamtho

There is still lots more wrong with 6:1:4 starting with needing tests, but this should help.

[schwehr]

https://github.com/schwehr/libais/commit/f2419b4f1ec699dafad4a903f15ca735c7f9198b is a better 6:1:4, but probably could still be better... e.g. handle if there is a non-standard amount of spare.

[schwehr]

If anyone has test messages that are crashing libais, please create a new issue with them.