search

ecrin-github / MDR_Aggregator

MIT License
1 stars 0 forks source link

Create Index fails because of `42P01: relation "core.new_search_objects" does not exist` #6

Open michelescarlato opened 2 months ago

michelescarlato commented 2 months ago

The Aggregator Create Index task fails due to:


30/08/2024 : 21:15 :   **** SETUP ****

30/08/2024 : 21:15 :   transfer data =  False
30/08/2024 : 21:15 :   create core =  False
30/08/2024 : 21:15 :   create json =  False
30/08/2024 : 21:15 :   do statistics =  False
30/08/2024 : 21:15 :   do iec =  False
30/08/2024 : 21:15 :   do indices =  True
30/08/2024 : 21:15 :   FTW tables recreated

30/08/2024 : 21:15 :   **** SETTING UP STUDY SEARCH TABLES ****

30/08/2024 : 21:15 :   **** CREATING JSON STUDY DATA ****

30/08/2024 : 21:15 :   **** UNHANDLED EXCEPTION ****

+++++++++++++++++++++++++++++++++++++++
30/08/2024 : 21:15 :   ***ERROR*** MDR_Aggregator application aborted

42P01: relation "core.new_search_objects" does not exist

POSITION: 76

   at Npgsql.Internal.NpgsqlConnector.<ReadMessage>g__ReadMessageLong|222_0(NpgsqlConnector connector, Boolean async, DataRowLoadingMode dataRowLoadingMode, Boolean readingNotifications, Boolean isReadingPrependedMessage)
   at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken)
   at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken)
   at Npgsql.NpgsqlDataReader.NextResult()
   at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken)
   at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken)
   at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior)
   at Npgsql.NpgsqlCommand.ExecuteDbDataReader(CommandBehavior behavior)
   at Dapper.SqlMapper.ExecuteReaderWithFlagsFallback(IDbCommand cmd, Boolean wasClosed, CommandBehavior behavior) in /_/Dapper/SqlMapper.cs:line 1066
   at Dapper.SqlMapper.QueryImpl[T](IDbConnection cnn, CommandDefinition command, Type effectiveType)+MoveNext() in /_/Dapper/SqlMapper.cs:line 1094
   at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
   at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
   at Dapper.SqlMapper.Query[T](IDbConnection cnn, String sql, Object param, IDbTransaction transaction, Boolean buffered, Nullable`1 commandTimeout, Nullable`1 commandType) in /_/Dapper/SqlMapper.cs:line 734
   at MDR_Aggregator.JSONStudyDataLayer.FetchObjectDetails(Int32 study_id) in C:\Users\ecrinadmin\RiderProjects\MDR_Aggregator\SearchStudyHelpers\JSONStudyDataLayer.cs:line 273
   at MDR_Aggregator.JSONStudyProcessor.CreateStudySearchResult(JSONFullStudy st) in C:\Users\ecrinadmin\RiderProjects\MDR_Aggregator\SearchStudyHelpers\JSONStudyProcessor.cs:line 427
   at MDR_Aggregator.SearchHelperJson.LoopThroughStudyRecords(Int32 offset) in C:\Users\ecrinadmin\RiderProjects\MDR_Aggregator\SearchHelpers\SearchHelperJSON.cs:line 95
   at MDR_Aggregator.CoreSearchBuilder.CreateJSONStudyData(Boolean create_table, Int32 offset) in C:\Users\ecrinadmin\RiderProjects\MDR_Aggregator\SearchHelpers\CoreSearchBuilder.cs:line 35
   at MDR_Aggregator.Aggregator.AggregateData(Options opts) in C:\Users\ecrinadmin\RiderProjects\MDR_Aggregator\TopLevelClasses\Aggregator.cs:line 244
   at Program.<Main>$(String[] args) in C:\Users\ecrinadmin\RiderProjects\MDR_Aggregator\TopLevelClasses\Program.cs:line 62
+++++++++++++++++++++++++++++++++++++++

30/08/2024 : 21:15 :   **** CLOSING LOG ****

image

Currently successfully re-run:

  1. Aggregate IEC data (source\repos\MDR_Aggregator\bin\Debug\net7.0\MDR_Aggregator.exe -l)

Still to be re-run:

  1. Aggregate Study and Object data (source\repos\MDR_Aggregator\bin\Debug\net7.0\MDR_Aggregator.exe -D)
  2. Create Aggregation Statistics (source\repos\MDR_Aggregator\bin\Debug\net7.0\MDR_Aggregator.exe -S)
  3. Create Core MDR DB (source\repos\MDR_Aggregator\bin\Debug\net7.0\MDR_Aggregator.exe -C)
  4. Create Indexes (this step) (source\repos\MDR_Aggregator\bin\Debug\net7.0\MDR_Aggregator.exe -X

Logs a. Aggregate IEC data (source\repos\MDR_Aggregator\bin\Debug\net7.0\MDR_Aggregator.exe -l) AG -I 2024-08-30 211744.log

michelescarlato commented 2 months ago

[conversation locked due to spam/phishing reasons]

michelescarlato commented 2 months ago

After re-run

two errors are there

30/08/2024 : 21:20 :   ----------------------------------------------------------------------
30/08/2024 : 21:20 :   AGGREGATING CTG DATA
30/08/2024 : 21:20 :   ----------------------------------------------------------------------

+++++++++++++++++++++++++++++++++++++++
30/08/2024 : 21:21 :   ***ERROR*** In ExecuteSQL; Exception while reading from stream, 
SQL was:  Insert into ad.study_iec_null (source_id, sd_sid, seq_num, iec_type_id, split_type, leader, indent_level,
        sequence_string, iec_text )
                             select 100120, t.sd_sid, seq_num, iec_type_id, split_type, leader, indent_level,
        sequence_string, iec_text   
                             from ctg_ad.study_iec_null t 
                             inner join ctg_ad.studies s
                             on t.sd_sid = s.sd_sid 
+++++++++++++++++++++++++++++++++++++++
30/08/2024 : 21:23 :   **** ADDING STUDY DATA ****

30/08/2024 : 21:23 :   Transferred 92518 study record data, from aggs DB to IEC DB, ids 1 to 100000
30/08/2024 : 21:23 :   Transferred 100000 study record data, from aggs DB to IEC DB, ids 100001 to 200000
30/08/2024 : 21:23 :   Transferred 19190 study record data, from aggs DB to IEC DB, ids 200001 to 300000
30/08/2024 : 21:23 :   Transferred 0 study record data, from aggs DB to IEC DB, ids 300001 to 400000
30/08/2024 : 21:23 :   Transferred 0 study record data, from aggs DB to IEC DB, ids 400001 to 500000
30/08/2024 : 21:23 :   Transferred 0 study record data, from aggs DB to IEC DB, ids 500001 to 600000
30/08/2024 : 21:23 :   Transferred 0 study record data, from aggs DB to IEC DB, ids 600001 to 700000
30/08/2024 : 21:23 :   Transferred 0 study record data, from aggs DB to IEC DB, ids 700001 to 800000
30/08/2024 : 21:23 :   Transferred 0 study record data, from aggs DB to IEC DB, ids 800001 to 900000
30/08/2024 : 21:23 :   Transferred 0 study record data, from aggs DB to IEC DB, ids 900001 to 1000000
30/08/2024 : 21:23 :   Transferred 0 study record data, from aggs DB to IEC DB, ids 1000001 to 1100000
30/08/2024 : 21:23 :   Transferred 0 study record data, from aggs DB to IEC DB, ids 1100001 to 1200000
30/08/2024 : 21:23 :   Transferred 0 study record data, from aggs DB to IEC DB, ids 1200001 to 1300000
30/08/2024 : 21:23 :   Transferred 0 study record data, from aggs DB to IEC DB, ids 1300001 to 1400000
30/08/2024 : 21:23 :   Transferred 0 study record data, from aggs DB to IEC DB, ids 1400001 to 1500000
30/08/2024 : 21:23 :   Transferred 0 study record data, from aggs DB to IEC DB, ids 1500001 to 1600000
30/08/2024 : 21:23 :   Transferred 0 study record data, from aggs DB to IEC DB, ids 1600001 to 1700000
30/08/2024 : 21:23 :   Transferred 0 study record data, from aggs DB to IEC DB, ids 1700001 to 1800000
30/08/2024 : 21:23 :   Transferred 0 study record data, from aggs DB to IEC DB, ids 1800001 to 1900000
30/08/2024 : 21:23 :   Transferred 0 study record data, from aggs DB to IEC DB, ids 1900001 to 2000000
30/08/2024 : 21:23 :   Transferred 99991 study record data, from aggs DB to IEC DB, ids 2000001 to 2100000
30/08/2024 : 21:23 :   Transferred 99984 study record data, from aggs DB to IEC DB, ids 2100001 to 2200000
30/08/2024 : 21:23 :   Transferred 99949 study record data, from aggs DB to IEC DB, ids 2200001 to 2300000
30/08/2024 : 21:23 :   Transferred 99926 study record data, from aggs DB to IEC DB, ids 2300001 to 2400000
30/08/2024 : 21:23 :   Transferred 92497 study record data, from aggs DB to IEC DB, ids 2400001 to 2500000
30/08/2024 : 21:23 :   Transferred 99839 study record data, from aggs DB to IEC DB, ids 2500001 to 2600000
30/08/2024 : 21:23 :   Transferred 99997 study record data, from aggs DB to IEC DB, ids 2600001 to 2700000
30/08/2024 : 21:23 :   Transferred 62151 study record data, from aggs DB to IEC DB, ids 2700001 to 2762213
30/08/2024 : 21:23 :   Transferred 966042 study record data, from aggs DB to IEC DB, in total

+++++++++++++++++++++++++++++++++++++++
30/08/2024 : 21:24 :   ***ERROR*** In ExecuteSQL; Exception while reading from stream, 
SQL was: update ad.studies k
               set display_title = s.display_title, 
               brief_description = s.brief_description, 
               iec_level_id = s.iec_level,               
               study_start_year = s.study_start_year, study_start_month = s.study_start_month, 
               study_type_id = s.study_type_id, study_enrolment = s.study_enrolment, 
               study_gender_elig_id = s.study_gender_elig_id, 
               min_age = s.min_age, min_age_units_id = s.min_age_units_id,
               max_age = s.max_age, max_age_units_id = s.max_age_units_id, 
               datetime_of_data_fetch = k.datetime_of_data_fetch
            from aggs_st.studies s
            where k.study_id = s.id  and k.id >= 1 and k.id < 25001 
+++++++++++++++++++++++++++++++++++++++

But the process terminated with Exit code 0.

AG -I 2024-08-30 211744.log

Re-running now.

michelescarlato commented 2 months ago

Second run only the update.ad.studies k appeared again.

31/08/2024 : 12:11 :   **** ADDING STUDY DATA ****

31/08/2024 : 12:11 :   Transferred 92518 study record data, from aggs DB to IEC DB, ids 1 to 100000
31/08/2024 : 12:12 :   Transferred 100000 study record data, from aggs DB to IEC DB, ids 100001 to 200000
31/08/2024 : 12:12 :   Transferred 22667 study record data, from aggs DB to IEC DB, ids 200001 to 300000
31/08/2024 : 12:12 :   Transferred 0 study record data, from aggs DB to IEC DB, ids 300001 to 400000
31/08/2024 : 12:12 :   Transferred 0 study record data, from aggs DB to IEC DB, ids 400001 to 500000
31/08/2024 : 12:12 :   Transferred 0 study record data, from aggs DB to IEC DB, ids 500001 to 600000
31/08/2024 : 12:12 :   Transferred 0 study record data, from aggs DB to IEC DB, ids 600001 to 700000
31/08/2024 : 12:12 :   Transferred 0 study record data, from aggs DB to IEC DB, ids 700001 to 800000
31/08/2024 : 12:12 :   Transferred 0 study record data, from aggs DB to IEC DB, ids 800001 to 900000
31/08/2024 : 12:12 :   Transferred 0 study record data, from aggs DB to IEC DB, ids 900001 to 1000000
31/08/2024 : 12:12 :   Transferred 0 study record data, from aggs DB to IEC DB, ids 1000001 to 1100000
31/08/2024 : 12:12 :   Transferred 0 study record data, from aggs DB to IEC DB, ids 1100001 to 1200000
31/08/2024 : 12:12 :   Transferred 0 study record data, from aggs DB to IEC DB, ids 1200001 to 1300000
31/08/2024 : 12:12 :   Transferred 0 study record data, from aggs DB to IEC DB, ids 1300001 to 1400000
31/08/2024 : 12:12 :   Transferred 0 study record data, from aggs DB to IEC DB, ids 1400001 to 1500000
31/08/2024 : 12:12 :   Transferred 0 study record data, from aggs DB to IEC DB, ids 1500001 to 1600000
31/08/2024 : 12:12 :   Transferred 0 study record data, from aggs DB to IEC DB, ids 1600001 to 1700000
31/08/2024 : 12:12 :   Transferred 0 study record data, from aggs DB to IEC DB, ids 1700001 to 1800000
31/08/2024 : 12:12 :   Transferred 0 study record data, from aggs DB to IEC DB, ids 1800001 to 1900000
31/08/2024 : 12:12 :   Transferred 0 study record data, from aggs DB to IEC DB, ids 1900001 to 2000000
31/08/2024 : 12:12 :   Transferred 99991 study record data, from aggs DB to IEC DB, ids 2000001 to 2100000
31/08/2024 : 12:12 :   Transferred 99984 study record data, from aggs DB to IEC DB, ids 2100001 to 2200000
31/08/2024 : 12:12 :   Transferred 99949 study record data, from aggs DB to IEC DB, ids 2200001 to 2300000
31/08/2024 : 12:12 :   Transferred 99926 study record data, from aggs DB to IEC DB, ids 2300001 to 2400000
31/08/2024 : 12:12 :   Transferred 92497 study record data, from aggs DB to IEC DB, ids 2400001 to 2500000
31/08/2024 : 12:12 :   Transferred 99839 study record data, from aggs DB to IEC DB, ids 2500001 to 2600000
31/08/2024 : 12:12 :   Transferred 99997 study record data, from aggs DB to IEC DB, ids 2600001 to 2700000
31/08/2024 : 12:12 :   Transferred 62151 study record data, from aggs DB to IEC DB, ids 2700001 to 2762213
31/08/2024 : 12:12 :   Transferred 969519 study record data, from aggs DB to IEC DB, in total

+++++++++++++++++++++++++++++++++++++++
31/08/2024 : 12:12 :   ***ERROR*** In ExecuteSQL; Exception while reading from stream, 
SQL was: update ad.studies k
               set display_title = s.display_title, 
               brief_description = s.brief_description, 
               iec_level_id = s.iec_level,               
               study_start_year = s.study_start_year, study_start_month = s.study_start_month, 
               study_type_id = s.study_type_id, study_enrolment = s.study_enrolment, 
               study_gender_elig_id = s.study_gender_elig_id, 
               min_age = s.min_age, min_age_units_id = s.min_age_units_id,
               max_age = s.max_age, max_age_units_id = s.max_age_units_id, 
               datetime_of_data_fetch = k.datetime_of_data_fetch
            from aggs_st.studies s
            where k.study_id = s.id  and k.id >= 1 and k.id < 25001 
+++++++++++++++++++++++++++++++++++++++

AG -I 2024-08-31 120650.log