Closed evacougnon closed 1 year ago
..
sorry @Chetnamann but still not working
in step 1 look for the collection named: "IMOS - Autonomous Underwater Vehicle (AUV) Facility"
there are no bounding boxes while selecting it and going to step 2 shows:
and that's version: Portal v4.42.104
Makes sense now, thanks
The bbox info comes from geonetwork, for example
have spatial extends in geonetwork
for those missing bbox in min-map above like this one af5d0ff9-bb9c-4b7c-a63c-854a630b6984, spatial extends is missing in geonetwork
@evacougnon Do you know if this field was created manually or generate by some process?
Further investigation found that the bbox will be update if the following query yield positive result. And I checked that if you run this query with the AUV harvester database, it return true.
// has any resource changed during the harvest
String changeQuery = ""
+ " select exists ("
+ " select 1"
+ " from " + <%=dbschema%> + ".indexed_file if"
+ " left join " + <%=dbschema%> + ".file_harvest fh on if.id = fh.file_id"
+ " where fh.last_change_run = (select last_run_no from " + <%=dbschema%> + ".index_job where name = ? )"
+ " and fh.harvest_type = ?"
+ " ) as changed"
This query stored in the iUpdateSpatialExtent_begin.javajet.
@evacougnon from the log the AUV data is having issue when last run @2020-03-06, after that no more log entries even the last update happens at 2022-02-16
2019-06-12 13:47:00;akGf5e;akGf5e;akGf5e;AUV;AUV_harvester;Default;6;Java Exception;iPostgresqlDbUpdate_4;liquibase.exception.MigrationFailedException:Migration failed for change set changelog.xml::change_type_wms_time_column::talend:
2019-06-12 13:47:02;40VlVU;40VlVU;40VlVU;AUV;AUV_harvester;Default;6;Java Exception;iPostgresqlDbUpdate_4;liquibase.exception.MigrationFailedException:Migration failed for change set changelog.xml::change_type_wms_time_column::talend:
2019-06-13 09:50:40;QMSEeG;QMSEeG;QMSEeG;AUV;AUV_harvester;Default;6;Java Exception;iPostgresqlDbUpdate_4;liquibase.exception.MigrationFailedException:Migration failed for change set changelog.xml::change_type_wms_time_column::talend:
2019-06-13 09:50:42;acHGt9;acHGt9;acHGt9;AUV;AUV_harvester;Default;6;Java Exception;iPostgresqlDbUpdate_4;liquibase.exception.MigrationFailedException:Migration failed for change set changelog.xml::change_type_wms_time_column::talend:
2020-03-06 12:39:28;c8NAUV;c8NAUV;c8NAUV;ARGO;Argo_harvester;Default;6;Java Exception;iUpdateIndex_1;java.lang.RuntimeException:liquibase.exception.LockException: Could not acquire change log lock. Currently locked by fe80:0:0:0:8f6:71ff:fec3:eaf9%ens3 (fe80:0:0:0:8f6:71ff:fec3:eaf9%ens3) since 3/5/20 11:18 PM;1
I suspect the harvester failed or changed so that the code to create the bounding box is not call.
Ignore this theory of error as I found log entry to proof this is not causing problem, this task run successfully and yet the job name quote the same.
***** completed finally block ***********
In finally block
INFO 2/28/23 1:39 PM:liquibase: Successfully acquired change log lock
INFO 2/28/23 1:39 PM:liquibase: Reading from srs_oc_soop_rad.databasechangelog
INFO 2/28/23 1:39 PM:liquibase: Reading from srs_oc_soop_rad.databasechangelog
INFO 2/28/23 1:39 PM:liquibase: Successfully released change log lock
* iUpdateIndex: FileResource[basePath: /mnt/ebs/tmp/SrsOcSoopRadHandlerlvt2ic06/temp/talend_basebo6evs5l, relativePath: IMOS/SRS/OC/radiometer/VMQ9273_Solander/2022/fv02-products/IMOS_SRS-OC_F_20221030T033446Z_VMQ9273_FV02_DALEC_END-20221030T073718Z.nc] is new or has been modified
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
* iNewOrModifiedFileList_3 modified/new resource file_id=998
* iNewOrModifiedFileList_1 modified/new resource file_id=998
* iUpdateSpatialExtent - files changed: true
* iUpdateSpatialExtent - updating spatial extent: true
* iUpdateSpatialExtent - updating uuid '28f8bfed-ca6a-472a-84e4-42563ce4df3f' from srs_oc_soop_rad.srs_oc_soop_rad_trajectory_map.geom using 3 degree resolution
log4j:WARN No appenders could be found for logger (org.apache.commons.httpclient.HttpClient).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
--- END TALEND OUTPUT ---
2023-02-28 13:39:15,122 INFO tasks.SRS_OC_SOOP_RAD[be98b87b-2d86-4c17-b475-026e66037bcd] updated file 'IMOS_SRS-OC_F_20221030T033446Z_VMQ9273_FV02_DALEC_END-20221030T073718Z_C-20230226T224944Z.nc': {'is_stored': True}
2023-02-28 13:39:15,123 SYSINFO tasks.SRS_OC_SOOP_RAD[be98b87b-2d86-4c17-b475-026e66037bcd] SrsOcSoopRadHandler transitioned to state: HANDLER_PUBLISHED
2023-02-28 13:39:15,123 SYSINFO tasks.SRS_OC_SOOP_RAD[be98b87b-2d86-4c17-b475-026e66037bcd] `postprocess` not overridden by child class, skipping step
2023-02-28 13:39:15,123 SYSINFO tasks.SRS_OC_SOOP_RAD[be98b87b-2d86-4c17-b475-026e66037bcd] SrsOcSoopRadHandler transitioned to state: HANDLER_POSTPROCESSED
2023-02-28 13:39:15,124 SYSINFO tasks.SRS_OC_SOOP_RAD[be98b87b-2d86-4c17-b475-026e66037bcd] get_notify_runner -> NotifyRunnerAdapter()
2023-02-28 13:39:15,124 SYSINFO tasks.SRS_OC_SOOP_RAD[be98b87b-2d86-4c17-b475-026e66037bcd] SrsOcSoopRadHandler transitioned to state: HANDLER_NOTIFIED_SUCCESS
2023-02-28 13:39:15,124 INFO tasks.SRS_OC_SOOP_RAD[be98b87b-2d86-4c17-b475-026e66037bcd] handler result for input_file '/mnt/ebs/processing/SRS_OC_SOOP_RAD/be98b87b-2d86-4c17-b475-026e66037bcd/IMOS_SRS-OC_F_20221030T033446Z_VMQ9273_FV02_DALEC_END-20221030T073718Z_C-20230226T224944Z.nc': SUCCESS
2023-02-28 13:39:15,124 SYSINFO tasks.SRS_OC_SOOP_RAD[be98b87b-2d86-4c17-b475-026e66037bcd] SrsOcSoopRadHandler transitioned to state: HANDLER_COMPLETED_SUCCESS
2023-02-28 13:39:15,132 SYSINFO tasks.SRS_OC_SOOP_RAD[be98b87b-2d86-4c17-b475-026e66037bcd] IncomingFileStateManager.state -> 'FILE_SUCCESS'
For this dataset https://portal-systest.aodn.org.au/search?uuid=78d588ed-79dd-47e2-b806-d39025194e7e
The log file shows the following lines
2022-08-15 12:30:52,867 SYSINFO tasks.SRS_ALTIMETRY[1caf673d-6f7d-4c1a-ad32-d47c9be349a4] executing /mnt/ebs/talend/jobs/srs_altimetry-srs_altimetry/java/SRS_ALTIMETRY_harvester/SRS_ALTIMETRY_harvester_run.sh --context_param paramFile="/mnt/ebs/talend/jobs/srs_altimetry-srs_altimetry/etc/srs_altimetry-srs_altimetry.conf" --context_param base=/mnt/ebs/tmp/SrsAltHandlerj39vqsw_/temp/talend_base1gff4i1i --context_param fileList=/mnt/ebs/tmp/SrsAltHandlerj39vqsw_/temp/talend_base1gff4i1i/file_listxn5f19as.txt --context_param logDir=/mnt/ebs/log/pipeline/harvest
2022-08-15 12:30:52,867 INFO tasks.SRS_ALTIMETRY[1caf673d-6f7d-4c1a-ad32-d47c9be349a4] --- START TALEND OUTPUT ---
updated file 'IMOS_SRSALT_TCPS_20011217T015051Z_BASJAS_FV01_SBE37d11m.nc': {'is_harvested': True}
INFO 8/15/22 12:30 PM:liquibase: Successfully acquired change log lock
INFO 8/15/22 12:30 PM:liquibase: Reading from srs_altimetry.databasechangelog
INFO 8/15/22 12:30 PM:liquibase: Reading from srs_altimetry.databasechangelog
INFO 8/15/22 12:30 PM:liquibase: Successfully released change log lock
* iUpdateIndex: FileResource[basePath: /mnt/ebs/tmp/SrsAltHandlerj39vqsw_/temp/talend_base1gff4i1i, relativePath: IMOS/SRS/ALTIMETRY/calibration_validation/SRSBASJAS/CTD_timeseries/IMOS_SRSALT_TCPS_20011217T015051Z_BASJAS_FV01_SBE37d11m.nc] is new or has been modified
Start of try block
tempFile_cid /tmp/talend-temp-8986280040131185496/changelog.xml
***********************
conn host:dbprod.emii.org.au, port:5432, schema:srs_altimetry, name:harvest?loginTimeout=1000&ssl=true&sslfactory=org.postgresql.ssl.NonValidatingFactory, user:srs_altimetry
url_cid:jdbc:postgresql://dbprod.emii.org.au:5432/harvest?loginTimeout=1000&ssl=true&sslfactory=org.postgresql.ssl.NonValidatingFactory
schema_cid:srs_altimetry
userName_cid:srs_altimetry
INFO 8/15/22 12:30 PM:liquibase: Successfully acquired change log lock
INFO 8/15/22 12:30 PM:liquibase: Reading from srs_altimetry.databasechangelog
INFO 8/15/22 12:30 PM:liquibase: Reading from srs_altimetry.databasechangelog
INFO 8/15/22 12:30 PM:liquibase: Successfully released change log lock
***** completed finally block ***********
In finally block
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
* iNewOrModifiedFileList_1 modified/new resource file_id=353
* iUpdateSpatialExtent - files changed: false
* iUpdateSpatialExtent - updating spatial extent: false
--- END TALEND OUTPUT ---
2022-08-15 12:31:11,947 INFO tasks.SRS_ALTIMETRY[1caf673d-6f7d-4c1a-ad32-d47c9be349a4] updated file 'IMOS_SRSALT_TCPS_20011217T015051Z_BASJAS_FV01_SBE37d11m.nc': {'is_stored': True}
The query that create the change / no change for is show below, I execute it in srs_altimetry schema of harvest db
select exists (
select 1
from indexed_file if
left join file_harvest fh on if.id = fh.file_id
where fh.last_change_run = (select last_run_no from index_job where name = 'index' )
and fh.harvest_type = 'data'
) as changed;
`
However it yield "true" when run it directly using sql console to read-only db of harvester, this means that it should execute the spatial extent code, so the error can either be the name or harvest_type argument during pipeline execution
I noticed that the two value is " in harvester setup, which mean its value is a string with ""
<elementParameter field="TEXT" name="JOBNAME" value=""index""/>
<elementParameter field="TEXT" name="HARVEST_TYPE" value=""data""/>
Did a workaround and create a utility tools to update the mini-map in geonetwork directly, code and how to use it in PR
@evacougnon is happy about the change, @vietnguyengit feel free to Dod it
will do
portal systest still offline
I think this kind of utility tool, given its nature related to generating information, should have Unit tests to guarantee data integrity/correctness if there are future updates to maintain the tool.
https://www.notion.so/imos-world/Definition-of-Done-DoD-b98281f003a44ce890dfba5af2f00337
I'll move this back to in progress for test-related criteria in DoD definition, need tests unless there are specific reasons to not have tests.
@vietnguyengit the testing should be very low priority, I doubt if it make sense to investigate time to do unit testing
@vietnguyengit the testing should be very low priority, I doubt if it make sense to investigate time to do unit testing
I will move this ticket to DoD because the effort to build unit tests seem to be unnecessary.
no changes I added to what @utas-raymondng did, which were satisfying so I DoDed this ticket.
This issue tracker is only for AODN Portal issues.
Steps to reproduce the issue
Expected behaviour
These collections should show a bounding box on step 1 based on their metadata record
Actual behaviour
There is no bounding box on the minimaps
Screenshots.
If the issue is related to the GUI, screenshots can be added to this issue via drag & drop.
What version of the aodn portal are you using?
4.42.102
Which browser are you using?
Chrome
Any extra information that might be useful in the debugging process.
Check with @atkinsn if that could come from their metadata record Associated test: https://aodn.testrail.com/index.php?/tests/view/84366
PR @craigrose
Dod @vietnguyengit