sqlite3.IntegrityError: NOT NULL constraint failed: ShipVendor.ship_id

[Tromador]

Do we need to do some manual adding of ships after all?

trade.py import -P eddblink -O clean -w
.
.
.
NOTE: Simultaneously processing UpgradeVendors, this will take quite a while.
# ABUKUNIN/Reilly Hub has been added:
# ALRAI/Bounds Hub has been added:
# ANGLII/Starzl Plant has been added:
# BETA TRIANGULI AUSTRALIS/Lunan Hub has been added:
# BETA TRIANGULI AUSTRALIS/Hutchinson City has been added:
# BANDJIGALI/Teng-hui Station has been added:
# 39 TAURI/Porta has been added:
# AGARLA/Wilson Port has been added:
# BETA TRIANGULI AUSTRALIS/Vinge Gateway has been added:
# APATHAAM/Reamy Dock has been added:
# BRANI/Virtanen Hub has been added:
# 85 PEGASI/Buckland Gateway has been added:
# BAGAI/Humphreys' Progress has been added:
# AIABIKO/Maxwell Orbital has been added:
# AIABIKO/Gooch Terminal has been added:
# BHUTAS/Feynman Terminal has been added:
# AMISI/Sargent Escape has been added:
# AMISI/Brooks' Folly has been added:
# 78 URSAE MAJORIS/Townshend Hub has been added:
# BRANI/Akiyama Market has been added:
# ALLOWA/Moore Ring has been added:
# 85 PEGASI/Borman Terminal has been added:
# ANLAVE/Suri Park has been added:
# 44 B OPHIUCHI/Feustel Gateway has been added:
# AULIN/Aulin Enterprise has been added:
# AMAHU/Kondratyev Orbital has been added:
# AMARAK/Vela Port has been added:
# ATFERO/Patry Dock has been added:
# BLODER/Szebehely Station has been added:
# BHUMLIAURT/Lichtenberg Port has been added:
# BD+69 530/Burbank Gateway has been added:
# 78 URSAE MAJORIS/Read Gateway has been added:
# BD+69 530/Leoniceno Orbital has been added:
# 31 AQUILAE/Russo Dock has been added:
# AMIJARA/Shepherd Outpost has been added:
# BIDIONO/Wellman Dock has been added:
# APAM NAPAT/Langley Hub has been added:
# AGUANA/Webb Dock has been added:
# ANLAVE/Bain Dock has been added:
# ANLAVE/Hogg City has been added:
# BD+55 1519/Arber Colony has been added:
# AMALANGKAN/Doi Landing has been added:
# AREXE/Janes Horizons has been added:
# BELU/Covey Gateway has been added:
# BINAR/Ferguson Dock has been added:
# BRANI/Barsanti Enterprise has been added:
# BAKONIRTARI/Gregory Vision has been added:
# BRANI/Noakes Port has been added:
# ANDRUNGARRI/Disch Port has been added:
# BETA-1 TUCANAE/Adams Orbital has been added:
# 42 AQUILAE/Born Station has been added:
# AVIK/Bruce Prospect has been added:
# ATLANTIS/Kimura Terminal has been added:
# 42 AQUILAE/Ride Market has been added:
# 78 URSAE MAJORIS/Teller Terminal has been added:
# 110 HERCULIS/Jahn Hangar has been added:
# BALTAH'SINE/Baltha'Sine Station has been added:
# AHAUT/Goldberg Orbital has been added:
# ALTAIR/Solo Orbiter has been added:
# 78 URSAE MAJORIS/Rominger Dock has been added:
# ANLAVE/Brunton Gateway has been added:
# ANLAVE/Kobayashi City has been added:
# ALCOR/Birmingham Dock has been added:
# BD-04 4138/Shunn Dock has been added:
# BRANGLAL/Gibson Horizons has been added:
# BD+47 1236/Dana Relay has been added:
# BPM 89444/Szameit Ring has been added:
# ALIOTH/Irkutsk has been added:
Traceback (most recent call last):
  File "D:\Games\Game Tools\Elite\Trade Dangerous\trade.py", line 104, in <module>
    main(sys.argv)
  File "D:\Games\Game Tools\Elite\Trade Dangerous\trade.py", line 77, in main
    results = cmdenv.run(tdb)
  File "D:\Games\Game Tools\Elite\Trade Dangerous\commands\commandenv.py", line 81, in run
    return self._cmd.run(results, self, tdb)
  File "D:\Games\Game Tools\Elite\Trade Dangerous\commands\import_cmd.py", line 124, in run
    if not plugin.run():
  File "D:\Games\Game Tools\Elite\Trade Dangerous\plugins\eddblink_plug.py", line 795, in run
    self.importStations()
  File "D:\Games\Game Tools\Elite\Trade Dangerous\plugins\eddblink_plug.py", line 430, in importStations
    modified))
  File "D:\Games\Game Tools\Elite\Trade Dangerous\plugins\eddblink_plug.py", line 95, in execute
    result = cur.execute(sql_cmd, args)
sqlite3.IntegrityError: NOT NULL constraint failed: ShipVendor.ship_id

[Tromador]

Ok - So I tried manually adding a Ship.csv and altered eddblink to not delete it and still no joy, so I guess that isn't it.

[eyeonus]

Why do I think this problem is being caused by the new ships that have been added recently, and that EDDB hasn't added them to the dumps yet?

[eyeonus]

Sorry, I meant EDCD,. I forgot the ships are gotten from a different place: https://raw.githubusercontent.com/EDCD/coriolis-data/master/dist/index.json

[eyeonus]

Can you find the line it's erroring on in stations.jsonl? That'd be a big help.

[Tromador]

I can try - but note that if you look in the coriolis data, Krait II and Challenger are both in there.

[Tromador]

{"id":487,"name":"Irkutsk","system_id":718,"updated_at":1531334552,"max_landing_pad_size":"L","distance_to_star":7784,"government_id":96,"government":"Democracy","allegiance_id":1,"allegiance":"Alliance","state_id":80,"state":"None","type_id":8,"type":"Orbis Starport","has_blackmarket":true,"has_market":true,"has_refuel":true,"has_repair":true,"has_rearm":true,"has_outfitting":true,"has_shipyard":true,"has_docking":true,"has_commodities":true,"import_commodities":["Cobalt","Rutile","Pyrophyllite"],"export_commodities":["Hydrogen Fuel","Aquaponic Systems","Resonating Separators"],"prohibited_commodities":["Narcotics","Combat Stabilisers","Imperial Slaves","Slaves","Personal Weapons","Battle Weapons","Toxic Waste","Bootleg Liquor","Landmines","Nerve Agents","Thargoid Scout Tissue Sample"],"economies":["High Tech"],"shipyard_updated_at":1531337538,"outfitting_updated_at":1531337538,"market_updated_at":1531337538,"is_planetary":false,"selling_ships":["Asp Explorer","Diamondback Explorer","Diamondback Scout","Fer-de-Lance","Orca","Python","Asp Scout","Beluga Liner","Dolphin","Alliance Chieftain"],"selling_modules":[778,779,780,781,782,798,799,800,801,802,808,809,810,811,812,813,814,815,816,817,824,826,827,828,829,830,831,832,835,838,839,841,842,844,845,846,847,848,853,856,857,858,862,863,865,872,873,874,875,876,878,879,880,882,884,885,886,887,888,889,890,891,892,893,894,896,897,898,899,900,905,909,915,926,927,928,929,930,931,932,933,934,935,938,939,943,944,945,947,948,950,951,956,957,958,960,961,962,963,965,966,967,968,970,971,972,973,974,975,976,977,978,979,981,982,983,984,985,996,997,998,999,1001,1002,1003,1005,1006,1007,1008,1009,1010,1013,1014,1015,1017,1018,1019,1022,1023,1024,1026,1028,1029,1030,1031,1032,1037,1038,1039,1040,1042,1043,1044,1045,1046,1047,1048,1050,1055,1065,1066,1068,1069,1070,1071,1072,1075,1076,1077,1078,1079,1080,1082,1083,1085,1086,1087,1088,1090,1091,1092,1093,1095,1096,1097,1098,1099,1101,1102,1103,1104,1107,1110,1113,1114,1115,1116,1117,1118,1119,1121,1122,1123,1124,1125,1129,1130,1134,1139,1140,1141,1144,1145,1146,1147,1148,1180,1181,1182,1183,1184,1185,1186,1187,1188,1189,1190,1191,1196,1197,1198,1199,1200,1201,1202,1203,1204,1205,1206,1207,1208,1209,1210,1212,1213,1214,1215,1216,1227,1230,1231,1233,1235,1238,1240,1241,1242,1243,1244,1245,1246,1251,1258,1259,1265,1266,1267,1274,1275,1281,1283,1286,1291,1295,1298,1299,1302,1303,1306,1308,1309,1310,1311,1313,1314,1315,1316,1317,1318,1319,1320,1321,1322,1323,1326,1327,1333,1334,1335,1339,1340,1341,1345,1346,1352,1353,1368,1369,1370,1371,1372,1373,1375,1379,1380,1381,1382,1383,1384,1385,1386,1387,1388,1398,1403,1407,1412,1413,1416,1417,1421,1423,1425,1428,1429,1432,1433,1434,1435,1436,1437,1438,1439,1440,1441,1442,1443,1444,1445,1446,1447,1448,1449,1450,1451,1452,1454,1455,1456,1457,1458,1459,1460,1503,1504,1505,1506,1507,1523,1524,1525,1526,1527,1528,1530,1531,1532,1534,1535,1536,1537,1541,1542,1543,1554,1555,1556,1557,1558,1560,1561,1562,1563,1564,1565,1566,1567,1569,1570,1577,1579,1581,1583,1584,1585,1586,1587,1588,1589,1590,1591,1592,1593,1596,1597,1598,1601,1603,1604,1605,1609,1612,1640,1641,1642,1643,1644],"settlement_size_id":null,"settlement_size":null,"settlement_security_id":null,"settlement_security":null,"body_id":10267,"controlling_minor_faction_id":1082}

[eyeonus]

If you do a level 3 debug, '-www', it'll say which line it's trying to insert when the error happens.

That will spit out a ton of data, obviously, so it might be easier to check the Ships.csv that EDDBlink generates against the list of ships being sold at the station it errored on.

IN the current dump on EDDB, this is the list at Irkutsk: "Asp Explorer","Diamondback Explorer","Diamondback Scout","Fer-de-Lance","Orca","Python","Asp Scout","Beluga Liner","Dolphin","Alliance Chieftain"

[Tromador]

Sure, except EDDBlink isn't making a Ship.csv, or hasn't got that far.

And it's line 487, as I posted above.

# ALIOTH/Irkutsk has been added:
# Inserting: 487, Irkutsk, 718, 7784, Y, L, Y, Y, 2018-07-11 18:42:32, Y, Y, Y, Y, N, 8
# ALIOTH/Irkutsk has shipyard, updating ships sold.
# ship_id:Asp Explorer,station_id:487,modified:2018-07-11 19:32:18
# ship_id:Diamondback Explorer,station_id:487,modified:2018-07-11 19:32:18
# ship_id:Diamondback Scout,station_id:487,modified:2018-07-11 19:32:18
# ship_id:Fer-de-Lance,station_id:487,modified:2018-07-11 19:32:18
# ship_id:Orca,station_id:487,modified:2018-07-11 19:32:18
# ship_id:Python,station_id:487,modified:2018-07-11 19:32:18
# ship_id:Asp Scout,station_id:487,modified:2018-07-11 19:32:18
# ship_id:Beluga Liner,station_id:487,modified:2018-07-11 19:32:18
# ship_id:Dolphin,station_id:487,modified:2018-07-11 19:32:18
# ship_id:Alliance Chieftain,station_id:487,modified:2018-07-11 19:32:18
Traceback (most recent call last):
  File "D:\Games\Game Tools\Elite\Trade Dangerous\trade.py", line 104, in <module>
    main(sys.argv)
  File "D:\Games\Game Tools\Elite\Trade Dangerous\trade.py", line 77, in main
    results = cmdenv.run(tdb)
  File "D:\Games\Game Tools\Elite\Trade Dangerous\commands\commandenv.py", line 81, in run
    return self._cmd.run(results, self, tdb)
  File "D:\Games\Game Tools\Elite\Trade Dangerous\commands\import_cmd.py", line 124, in run
    if not plugin.run():
  File "D:\Games\Game Tools\Elite\Trade Dangerous\plugins\eddblink_plug.py", line 795, in run
    self.importStations()
  File "D:\Games\Game Tools\Elite\Trade Dangerous\plugins\eddblink_plug.py", line 430, in importStations
    modified))
  File "D:\Games\Game Tools\Elite\Trade Dangerous\plugins\eddblink_plug.py", line 95, in execute
    result = cur.execute(sql_cmd, args)
sqlite3.IntegrityError: NOT NULL constraint failed: ShipVendor.ship_id

[eyeonus]

Right, yes, it doesn't export the csv files until it's done importing. You'll have to look at the Ship table in the DB directly then.

Actually, if you run '-O ship', it'll only do import for the ships, so it will generate the Ship.csv ....

Obviously it's failing on the Chieftain, although that makes no sense.

[Tromador]

Nothing in the database, I guess it hasn't hit a "commit" at this stage in the import.

[eyeonus]

Okay, do a run with just the 'ship' option, nothing else, except maybe 'force'.

[Tromador]

ship,force has the exact same result.

I notice that in the Coriolis JSON file, the

    "eddbID": 33,

is the same for both Alliance ships, would that cause it to screw up?

[eyeonus]

It would. That's supposed to be unique. EDCD messed up somewhere, I think.

[eyeonus]

EDDB doesn't even have the challenger. Arg.

[eyeonus]

Well, at least we know where the error is coming from. EDDB hasn't updated with the new ships.

[eyeonus]

The Python and the Krait have the same EDDB ID as well. Arg. I'll put in a temp fix for now.

[Tromador]

I've left a message on (what I hope is) their discord.

[eyeonus]

Hmm. According to the dump, there aren't any stations selling the Krait MkII.

[Tromador]

Probably just nobody has bothered to upload the data yet, or something. Still, that doesn't mean that the support for it should be broken.

And nobody is answering me on their discord, so *big shrug*

[eyeonus]

I've added special handling for those two ships to the plugin. Both solo and eye-TD versions, so this shouldn't be an issue anymore. Update and run, let me know.

[Tromador]

Yup. I see what you did. Makes sense. Hopefully they will sort out a permanent fix.

Running now and it's moved on to doing upgrade vendors, so looking good. Assuming it completes, I'll close this query.

[eyeonus]

Well, I put in issues on the EDCD github and sent an email asking EDDB.io to add the ships using EDDB's contact us form, so hopefully something will be done. In the meantime.... 🛌

[eyeonus]

"Why do I think this problem is being caused by the new ships that have been added recently, and that EDDB hasn't added them to the dumps yet?"

I hate it when I'm right. :)

[Tromador]

No you don't. You love it. Closing :)