Data Import Issue

[tomtom5152]

Originally reported by: David Taylor (BitBucket: davidtaylorhq, GitHub: davidtaylorhq)

---

During the import from Postec RIGS, a missing condition in an SQL statement (here) has resulted in items originally associated with "quotes" being associated with "events" in the new system.

The items can be identified in the old database using the SQL statement:

#!SQL

SELECT DISTINCT r.id
FROM rig_items AS i
INNER JOIN eventdetails AS e ON i.eventdetail_id = e.id
INNER JOIN rigs AS r ON e.describable_id = r.id
WHERE e.describable_type !=  'Rig'

The solution is to delete all of these items from the new system. This will return all events to the correct state. This should be done while adding a revision to the reversion history.

A one-time script to do this is:

#!python

from RIGS import models
import reversion

listOfItems = [92,94,96,98,99,100,143,144,145,201,153,154,166,219,218,234,221,222,235,273,274,300,332,333,680,681,682,683,684,706,395,396,397,398,399,616,617,492,493,496,659,519,528,529,539,599,600,559,608,607,735,734,741,829,830,831,832,833,834,835,836,837,838,653,662,664,667,719,722,762,763,776,777,778,779,780,781,909,908,910,911,793,794,795,845,873,874,898,896,895,879,923,924,925,927,928,929,930,931,980,988,990,991,998,999,1000,1001,1002,1003,1018,1019,1020,1093,1094,1126,1127,1128,1135,1215,1180,1214,1216,1190,1191,1600,1601,1602,1603,1604,1605,1606,1229,1230,1231,1232,1238,1239,1270,1256,1257,1271,1344,1345,1342,1343,1346,1300,1302,1303,1329,1330,1331,1332,1333,1334,1403,1370,1378,1441,1466,1467,1468,1470,1471,1475,1455,1454,1456,1452,1499,1500,1501,1579,1580,1581,1582,1583,1647,1648,1649,1685,1686,1687,1688,1689,1678,1712,1713,1714,1758,1759,1760,1761,1855,1723,1770,1771,1820,1821,1822,1965,1966,1952,1954,1951,1953,2033,2034,2035,2036,2061,2105,2138,2140,2162,2160,2161,2159,2144,2135,2143,2379,2380,2381,2382,2383,2384,2222,2223,2224,2225,2226,2272,2273,2274,2275,2276,2277,2278,2279,2280,2281,2282,2283,2284,2285,2286,2302,2303,2304,2305,2306,2307,2308,2309,2310,2311,2312,2313,2314,2315,2316,2317,2318,2319,2320,2321,2322,2323,2324,2325,2326,2327,2328,2329,2330,2331,2332,2333,2334,2335,2336,2337,2338,2339,2340,2341,2342,2343,2344,2345,2346,2347,2348,2349,2350,2351,2352,2353,2354,2355,2356,2357,2358,2359,2360,2361,2439,2428,2440,2423,2492,2442,2443,2444,2445,2446,2447,2448,2449,2450,2451,2452,2453,2454,2455,2456,2457,2458,2461,2462,2463,2464,2501,2502,2503,2504,2505,2506,2507,2508,2509,2706,2707,2708,2709,2710,2926,2927,2928,2929,2930,2931,2932,2963,2966,2967,2981,2982,2983,3102,3103,3104,3105,3106,3107,3108,3109,3110,3111,3112,3113,3114,3043,3044,3045,3046,3047,3048,3049,3050,3051,3052,3053,3054,3055,3140,3137,3138,3139,3143,3213,3214,3222,3228,3223,3221,3225,3229,3242,3243,3484,3611,3486,3487,3488,3489,3490,3492,3505,3506,3507,3508,3509,3510,3634,3641,3640,3648,3649,3678,3679,3680,3682,3767,3768,3823,3824,3819,3788,3821,3878,3883,3916,3871,3915,3881,3894,3917,3864,3899,4172,4174,4179,4180,4065,4066,4001,4002,4003,4004,4005,4006,4007,4040,4041,4042,4112,4113,4115,4116,4117,4119,4120,4122,4123,4124,4299,4297,4324,4421,4420,4419,4418,4388,4389,4417,4660,4711,4712,4723,4724,4832,4830,4831,4828,4829,4801,4827,4839,4840,4841,4842,4843,4844,4845,4846,4847,4848,4849,4850,4851,4852,5156,5157,5158,4878,4879,4880,4906,4882,4888,4919,4936,4921,4923,4924,4925,4930,4931,4932,4933,4976,4977,4978,5051,5052,5053,5054,5024,5027,5028,5254,5310,5309,5308,5313,5312,5297,5298,5299,5300,5395,5396,5397,5398,5399,5400,5815,5816,5817,5796,5797,5808,5809,5810,5811,5812,5868,5869,5925,5926,6227,6230,6229,6228,6231,6232,6233,6234,6235,6236,6237,6453,6454,6455,6456,6252,6251,6244,6245,6168,6246,6658,6657,6328,6656,6659,6660,6661,6662,6663,6664,6665,6666,6668,7002,6261,6262,6263,6264,6265,6266,6267,6725,6377,6378,6472,6492,6491,6487,6490,6493,6494,6654,6655,6849,6850,6851,6860,6861,6862,6863,7006,7102,7008,7107,7092,7101,7105,7013,7108,7104,7018,7019,7020,7021,7191,7192,7194,7195,7196,7197,7198,7199,7200,7204,7205,7206,7692,7693,7694,7695,7754,7755,7756,7757,7758,7759,8000,8536,8537,8538,8588,8589,8590,8592,8593,8594,8774,8773,8775,8770,8771,8772,8903,8904,8905,8906,8907,8908,8909,8910,8911,8912,8913,8914,8915,8916,8917,8918,8959,8960,8961,8962,8963,8964,8965,8966,8967,8968,8969,8970,8971,8972,8973,8974,8975,8976,8977,8978,8979,8980,8981,8982,8983,8984,8985,8986,8987,8988,8989,8990,8991,9021,9022,9058,9059,9195,9196,9281,9286,9287,9288,9289,9425,9428,9427,9846,9847,9848,9905,9906,9907,9908,9909,9910,9911,9912,9914,9915,9916,9917,9918,9919,9920,9921,9922,9923,9933,9934,9935,10166,10167,10168,10169,10170,10171,10172,10173,9994,9995,9996,9997,9998,9999,10000,10001,10002,10003,10004,10022,10023,10024,10025,10026,10027,10085,10034,10087,10082,10086,10071,10072,10073,10074,10079,10075,10076,10077,10078,10080,10096,10097,10098,10099,10100,10101,10102,10103,10110,10111,10112,10113,10114,10115,10116,10117,10118,10161,10165,10163,10222,10220,10221,10219,10234,10235,10296,10295,10392,10675,10676,10783,10897,10892,10889,10894,10895,10896,10961,10962,11078,11079,11120,11121,11125]

items = models.EventItem.objects.filter(pk__in=listOfItems)
eventspk = list(items.values_list('event', flat=True).distinct())
events = []

events = list(models.Event.objects.filter(pk__in=eventspk))

items.delete()

for event in events:
    with reversion.create_revision():
        event.save(force_update=True)

This can be run using manage.py shell

Currently this throws ValidationErrors on event.save() due to issue #156

---

• Bitbucket: https://bitbucket.org/nottinghamtec/pyrigs/issue/157

[tomtom5152]

Whilst this still exists, it hasn't been an issue at all => closed

[davidtaylorhq]

I disagree, there is blatantly incorrect data in RIGS. People trust past events, and use them as a basis for planning future events. If you're happy for me to run the above script, it can be fixed in about 10 seconds

[tomtom5152]

Is that not just deleting the quotes, surely even if they weren't rigs, the info may still be of some use?

[davidtaylorhq]

@tomtom5152 The info could be useful, but right now it's associated with random rigs. EventItems from quote pk=7 have ended up associated with rig pk=7 So you could potentially end up with WOF kit ending up associated with an open mic night

[tomtom5152]

Ah ok I misunderstood the issue. In that case, if you are confident that script is correct and you have appropriate snapshots etc, feel free to run it.

[davidtaylorhq]

Did this thing