[From #2943] regex failure

[MikhailKasimov]

Hello!

Resounding #2943 : https://github.com/stamparm/maltrail/issues/2943#issuecomment-515182216

for further investigation/testings.

[netstat-peanut]

I'm seeing what appears to be another regex issue. The error is not the same but I don't want to open a new issue without checking in here.

sensor.py dies when attempting to update trails for the first time post-launch. This error as lists are being downloaded:

Exception in thread Thread-2: Traceback (most recent call last): File "/usr/local/lib/python2.7/threading.py", line 801, in __bootstrap_inner self.run() File "/usr/local/lib/python2.7/threading.py", line 1073, in run self.function(*self.args, **self.kwargs) File "/usr/local/share/maltrail/core/parallel.py", line 67, in update_timer trails.update(_) File "/usr/local/share/maltrail/core/trailsdict.py", line 55, in update setattr(self, attr, getattr(value, attr)) AttributeError: 'TrailsDict' object has no attribute '_regex'

And this error when they're being processed: Exception in thread Thread-1: Traceback (most recent call last): File "/usr/local/lib/python2.7/threading.py", line 801, in __bootstrap_inner self.run() File "/usr/local/lib/python2.7/threading.py", line 1073, in run self.function(*self.args, **self.kwargs) File "/usr/local/share/maltrail/sensor.py", line 769, in update_timer if "static" in trails[trail][1]: File "/usr/local/share/maltrail/core/trailsdict.py", line 78, in __getitem__ return (self._infos[int(_[0])], self._references[int(_[1])]) IndexError: list index out of range

[stamparm]

@netstat-peanut please update and retry with the latest revision/commit. Patched couple of related bugs

[netstat-peanut]

Thank you, @stamparm , for your stellar response time!

I think you may have taken a step in the right direction here. Previously, I would see numerous "[x] something went wrong during remote data retrieval" errors during the update process, after which the above error ending in "_regex" would appear, concluding with the "list index out of range" error at the end of the trail update process.

Now after updating, I only received one "[x] something went wrong during remote data retrieval" error during updating (re: torstatus.py), not a single "_regex" error, but the sensor process still kicked out with "list index out of range" at the end.

Subsequently, I moved torstatus.py to torstatus.py.bak, re-ran the process with short update time of 150, and the resulting update had zero "[x] something went wrong during remote data retrieval", and no "_regex" error, however the process died with "list index out of range".

I hope this isn't too confusing. Let me know if I can help with additional details!

EDIT: Here is the specific error [i] post-processing trails (this might take a while)... [i] update finished [i] trails stored to '/root/.maltrail/trails.csv' Exception in thread Thread-1: Traceback (most recent call last): File "/usr/local/lib/python2.7/threading.py", line 801, in __bootstrap_inner self.run() File "/usr/local/lib/python2.7/threading.py", line 1072, in run self.function(*self.args, **self.kwargs) File "/usr/local/share/maltrail/sensor.py", line 772, in update_timer if "static" in trails[trail][1]: File "/usr/local/share/maltrail/core/trailsdict.py", line 82, in __getitem__ return (self._infos[int(_[0])], self._references[int(_[1])]) IndexError: list index out of range

[mimugmail]

@stamparm can you also Tag a new version of this fix it? We also have user reports from OPNsense users related to this.

My own firewall at home also has very high memory usage since around one week, not sure If this is also belonging to this one.

[stamparm]

@netstat-peanut can you please retry with the latest revision? @mimugmail I will tag it once the issue is solved. Also, there were no memory-related changes for some time

[netstat-peanut]

@stamparm Changes to def clear(self): in trailsdict.py appear to have fixed the problem. I had a few [x] something went wrong during remote data retrieval errors during the updates thus far (both from data.netlab.360.com) but I'm not convinced they're not related to a temporal issue with name resolution / connectivity in my environment.

Question: should stdout go from "update process -> post-processing trails -> update finished -> trails stored" OR should "trails stored" be followed by [o] running...? In my case stdout ends at [i] trails stored before rolling to another update (still timed at 150 for testing purposes), and I have not verified whether or not it's actually "running" in the interim.

It's now gone through 3 post-initialization update routines without dying. :)

[stamparm]

@netstat-peanut current stdout output should be ok(ish). [o] running... runs from that first message until program dies. I've just double checked and it really works as expected, even after X updates

[stamparm]

@mimugmail tagged this moment the 0.16 (https://github.com/stamparm/maltrail/releases/tag/0.16)

[mimugmail]

.. and landed in FreeBSD https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=241736 .. thx 👍

[netstat-peanut]

thanks @stamparm and @mimugmail !!