Foreign Key constraint on ingest?

wfastrononomer commented 2 months ago

" # 2024-09-02 13:49:09.52: [39441] Ingest file left on file share: //unas/ samba/share/cu34id39441_unas-1_testVSAnjcUVDR6_casu_ultravistaMapRemeasurementRa w.dat1 " # 2024-09-02 13:49:09.52: mx.ODBC.Error.IntegrityError: The INSERT state ment conflicted with the FOREIGN KEY constraint "fk_ultravistaMapRemeasurementRa w_catalogueID_to_MapFrameStatus_catalogueID". The conflict occurred in database "testVSAnjcUVDR6", table "dbo.MapFrameStatus", column 'catalogueID'. Script executed: BULK INSERT ultravistaMapRemeasurementRaw FROM '\UNAS\share\cu 34id39441_unas-1_testVSAnjcUVDR6_casu_ultravistaMapRemeasurementRaw.dat1' WITH ( firstrow=1, datafiletype='native', tablock, check_constraints, order(catalogueID ,apertureID))

Traceback (most recent call last): File "/unas/njc/VDFS/ListDrivenClean/src/curation/wsatools/DbConnect/IngIngest er.py", line 511, in tryIngest ingestCU(ingLogFile, histInfo, ingestOrder, fileInfo) File "/unas/njc/VDFS/ListDrivenClean/src/curation/wsatools/DbConnect/IngIngest er.py", line 479, in ingestCU4 isCu3=False, cuEventID=ingLogFile.cuEventID) File "/unas/njc/VDFS/ListDrivenClean/src/curation/wsatools/DbConnect/IngIngest er.py", line 872, in ingest deleteFile=False, isOrdered=not isCu3)

wfastrononomer commented 2 months ago

How many ingest files haven't been ingested and why - let's convert this to a csv and find which catalogueIDs are missing?

np.unique(rawData['catalogueID'])

1339 1918 2321 2561 2995 3162 3220 3700 All of these have entries in MapFrameStatus! What is going wrong. Try ingesting again? Which log files associated - grep Redid this for 34991 - on all tables - but none missing in MapFrameStatus or MapCatalogueDetector. Column name='catalogueID' dtype='int64' length=4> 1454 2264 2624 3582 >>> astroData = Table.read('/unas/samba/share/cu34id34991_unas-1_testVSAnjcUVDR6_casu_ultravistaMapRemeasurementAstrometry.dat1.csv') >>> np.unique(astroData['catalogueID']) 1454 2264 2624 3582 >>> photoData = Table.read('/unas/samba/share/cu34id34991_unas-1_testVSAnjcUVDR6_casu_ultravistaMapRemeasurementPhotometry.dat1.csv') >>> np.unique(photoData['catalogueID']) 1454 2264 2624 3582

wfastrononomer commented 2 months ago

Check places like: /mnt/ramses24/cu34id39441_unas-1_testVSAnjcUVDR6_casu.log

--> /mnt/ramses24/cu34id39441_unas-1_testVSAnjcUVDR6_casu.log.failed

Check failed logs.

Reran first --> same error, but why?

Ingesting csv file using Ingester.ingestTable works fine!!

Also ingesting binary file using Ingester.ingestTable works fine!!

Is this an issue with IngestCuFiles?? Overly complicated?

I should replace extp.run(IngestCuFiles) with Ingester.ingestTable() calls - but need to know which binary files.

Also should replace some of extp.run(DataBuilder)

wfastrononomer commented 1 month ago

select rolledBack,COUNT(*) from ArchiveCurationHistory as a,ProgrammeCurationHistory as p where a.cuEventID=p.cuEventID and a.cuID=34 and p.programmeID=160 and a.comment like 'failed:%' group by rolledBack

| | rolledBack | | |

|>| 1 | 758 |<|

But is this the case for after first part. Could compare contents of /mnt/ramses24/cu34id. and this

wfastrononomer commented 1 month ago

5 files don't ingest. First 5. But when I randomly tried some other files before they did. Have I made a mistake with set up. Why did my tester work, but this code didn't?

" # 2024-09-08 11:29:51.13: Selected 5 binary files to ingest " # 2024-09-08 11:29:51.13: Checking database schema for ingest tables... " # 2024-09-08 11:29:54.29: [32971] Ingesting into ultravistaMapRemeasurementAstro metry... " # 2024-09-08 11:29:54.50: [32971] Ingest file left on catalogue server: \ \CU34ID32971_UNAS-1_TESTVSANJCUVDR6_CASU_ULTRAVISTAMAPREMEASUREMENTASTROMETRY.DA T1\ " # 2024-09-08 11:29:54.50: str: cu34id32971_unas-1_testVSAnjcUVDR6_casu_ul travistaMapRemeasurementAstrometry.dat1 did not ingest into ultravistaMapRemeasu rementAstrometry " # 2024-09-08 11:29:54.50: [39611] Ingesting into ultravistaMapRemeasurementRaw.. . " # 2024-09-08 11:29:54.70: [39611] Ingest file left on catalogue server: \ \CU34ID39611_UNAS-1_TESTVSANJCUVDR6_CASU_ULTRAVISTAMAPREMEASUREMENTRAW.DAT1\ " # 2024-09-08 11:29:54.70: str: cu34id39611_unas-1_testVSAnjcUVDR6_casu_ul travistaMapRemeasurementRaw.dat1 did not ingest into ultravistaMapRemeasurementR aw

This ingests fine using CheckMapRemeasurementIngests - why?

" # 2024-09-08 11:29:54.70: [37881] Ingesting into ultravistaMapRemeasurementAstrometry... " # 2024-09-08 11:29:54.91: [37881] Ingest file left on catalogue server: \CU34ID37881_UNAS-1_TESTVSANJCUVDR6_CASU_ULTRAVISTAMAPREMEASUREMENTASTROMETRY.DAT1\ " # 2024-09-08 11:29:54.91: str: cu34id37881_unas-1_testVSAnjcUVDR6_casu_ultravistaMapRemeasurementAstrometry.dat1 did not ingest into ultravistaMapRemeasurementAstrometry " # 2024-09-08 11:29:54.91: [31151] Ingesting into ultravistaMapRemeasurementAstrometry... " # 2024-09-08 11:29:55.11: [31151] Ingest file left on catalogue server: \CU34ID31151_UNAS-1_TESTVSANJCUVDR6_CASU_ULTRAVISTAMAPREMEASUREMENTASTROMETRY.DAT1\ " # 2024-09-08 11:29:55.11: str: cu34id31151_unas-1_testVSAnjcUVDR6_casu_ultravistaMapRemeasurementAstrometry.dat1 did not ingest into ultravistaMapRemeasurementAstrometry " # 2024-09-08 11:29:55.11: [5271] Ingesting into ultravistaMapRemeasurementRaw... " # 2024-09-08 11:29:55.32: [5271] Ingest file left on catalogue server: \CU34ID5271_UNAS-1_TESTVSANJCUVDR6_CASU_ULTRAVISTAMAPREMEASUREMENTRAW.DAT1\ " # 2024-09-08 11:29:55.32: str: cu34id5271_unas-1_testVSAnjcUVDR6_casu_ultravistaMapRemeasurementRaw.dat1 did not ingest into ultravistaMapRemeasurementRaw

wfastrononomer commented 1 week ago

It seems that setting isOrdered=True causes the issues with foreign key constraint. I am not sure why though.

wfau / ScienceArchives