Listener fails to update some stations

[eyeonus]

For some reason, the database isn't being updated by the listener for some stations when processing the EDDN messages.

First step is to add some debug output to track down the source of the problem.

[bgol]

Found one more:

            elif config['verbose']:
                print("ERROR: Not found in Stations: " + system + "/" + station)
                continue

should be

            else:
                if config['verbose']:
                    print("ERROR: Not found in Stations: " + system + "/" + station)
                continue

[bgol]

If you're running in WAL journal there is no need to stop the updater while exporting. WAL allowes multiple reader and one writer. Only the EDDB update can't run at the same time.

[Tromador]

That's part of the learning curve I think Bernd. It's only running WAL on my server, not as standard. Once I'm happy with the server tunings I'm playing with, I may well send a PR to the TD source to update the SQL schema with a nifty collection of PRAGMA, but until then we need to assume that users are NOT running WAL mode. (Though in theory won't be running in server mode either and thus not exporting, but anyway).

[aadler]

Oh My. One to two orders of magnitude faster than the client vesion, something like four orders of magnitude faster than the old server version. Yowzers. Well done @bgol !!

[bgol]

The EDDBlink-listener should not be run by end-users at all! There shouldn't be to many EDDN-Listeners running.

[aadler]

So, @bgol, it was a combination of minimizing the number of transactions and preparing the SQL statements that made this so much faster?

[Tromador]

Bernd, I agree.

The history of this project starts with the death of maddavo's TD services. If my hosting were to suddenly explode, then again there is no TD server. One of the goals is to ensure that in the event of no server, users can pick up TD and use it effectively - so we want the listener to work correctly out of the box, even if we might normally discourage casual use.

(and this may be a discussion for the forum, not a github ticket :) )

[bgol]

@aadler It was mostly the change of transaction handling. I think the change to executemany() doesn't make much of a difference, but I didn't test that.

[eyeonus]

I was only asking for an example SQL statement, as in an example of what I should be trying to build in the code, but doing all the work for me works too. :)

[bgol]

I just was in a programming mood. Don't expect it too often. :)

[eyeonus]

No worries. Appreciated all the same.

[eyeonus]

Well, this shouldn't happen....

Market update for SEMALI/RASCH HUB finished in 0.516 seconds.
Exception in thread Thread-3:
Traceback (most recent call last):
  File "C:\Python37\lib\threading.py", line 917, in _bootstrap_inner
    self.run()
  File "C:\Python37\lib\threading.py", line 865, in run
    self._target(*self._args, **self._kwargs)
  File "S:\Elite Dangerous Programs\Trade Dangerous\eddblink_listener.py", line 649, in process_messages
    curs.executemany(insStmt, itemList)
sqlite3.IntegrityError: UNIQUE constraint failed: StationItem.station_id, StationItem.item_id

[Tromador]

Same in the server logs on that station.

[Tromador]

With the additional when I shut it down:

Exception in thread Thread-4:
Traceback (most recent call last):
  File "/home/elite/local/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/home/elite/local/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "./eddblink_listener.py", line 724, in export_listings
    results = list(fetchIter(db_execute(db, "SELECT * FROM StationItem WHERE from_live = 1 ORDER BY station_id, item_id")))
  File "./eddblink_listener.py", line 665, in fetchIter
    results = cursor.fetchmany(arraysize)
AttributeError: 'NoneType' object has no attribute 'fetchmany'

[Tromador]

On reflection - it could be that a bad schema was sent. As there isn't any error trapping, it's hard to know.

Perhaps we should code to say

if error
    log the schema
    carry on regardless

[eyeonus]

Yeah, I'm fairly certain it's not that station, it's the station immediately after it, whatever that was.

On Wed, Jul 25, 2018, 02:43 Stefan Morrell notifications@github.com wrote:

Same in the server logs on that station.

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/eyeonus/EDDBlink-listener/issues/7#issuecomment-407680426, or mute the thread https://github.com/notifications/unsubscribe-auth/ABpz4NrMQYgkTPdVfqA6T0vr67VIFIB8ks5uKC-_gaJpZM4VL3zU .

[Tromador]

Yes - that would make sense.

That doesn't change my proposal though - trap the error, log the offending data, carry on to the next one.

Just that we both dropped on the same station and it's otherwise been ok, suggests to me it was garbage data, rather than a bug at our end and as such we need to make provision for such errors.

[bgol]

Yeah, I'm fairly certain it's not that station, it's the station immediately after it, whatever that was.

It was EDDI https://ross.eddb.io/eddn/log/75839746 HeliostaticFurnaces is listed twice.

EDDI does send wrong commodity messages since 3.0.1-X. They only corrected it recently with version 3.0.1-rc5 I think. EDDB has it blacklisted.

That's the downside of the executemany. In my own code I keep track of the IDs and throw a warning if a item comes more than one time, eg:

unqItems = set()
for commodity in commodities:
     ...
     if item_id in unqItems:
         if config['verbose']:
             print("Multiple item '" + itemName + "'")
         continue
     unqItems.add(item_id)
     ....