search

aodn / aodn-portal

AODN Open Geospatial Portal
https://portal.aodn.org.au/
GNU General Public License v3.0
21 stars 13 forks source link

Geographical bounding box not shown on all minimaps from step 1 #2957

Closed evacougnon closed 1 year ago

evacougnon commented 1 year ago

This issue tracker is only for AODN Portal issues.

Steps to reproduce the issue

  1. Step 1 of Portal systest https://portal-systest.aodn.org.au/search
  2. Look for the following collections:

Expected behaviour

These collections should show a bounding box on step 1 based on their metadata record

Actual behaviour

There is no bounding box on the minimaps

Screenshots.

If the issue is related to the GUI, screenshots can be added to this issue via drag & drop.

What version of the aodn portal are you using?

4.42.102

Which browser are you using?

Chrome

Any extra information that might be useful in the debugging process.

Check with @atkinsn if that could come from their metadata record Associated test: https://aodn.testrail.com/index.php?/tests/view/84366

PR @craigrose

  1. https://github.com/aodn/utilities/pull/272
  2. https://github.com/aodn/utilities/pull/274
  3. https://github.com/aodn/utilities/pull/275
  4. https://www.notion.so/imos-world/Bound-box-missing-on-Mini-map-5eee7b9d84ba420a961c668edbb28e61

    Dod @vietnguyengit

Chetnamann commented 1 year ago

..

evacougnon commented 1 year ago

sorry @Chetnamann but still not working

in step 1 look for the collection named: "IMOS - Autonomous Underwater Vehicle (AUV) Facility" Screenshot from 2023-05-25 13-43-32

there are no bounding boxes while selecting it and going to step 2 shows: Screenshot from 2023-05-25 13-45-36

and that's version: Portal v4.42.104

Chetnamann commented 1 year ago

Makes sense now, thanks

utas-raymondng commented 1 year ago

The bbox info comes from geonetwork, for example

image

have spatial extends in geonetwork

for those missing bbox in min-map above like this one af5d0ff9-bb9c-4b7c-a63c-854a630b6984, spatial extends is missing in geonetwork

image

@evacougnon Do you know if this field was created manually or generate by some process?

utas-raymondng commented 1 year ago

Further investigation found that the bbox will be update if the following query yield positive result. And I checked that if you run this query with the AUV harvester database, it return true.

    // has any resource changed during the harvest
    String changeQuery = ""
      + " select exists ("
      + "   select 1"
      + "   from " + <%=dbschema%> + ".indexed_file if"
      + "   left join " + <%=dbschema%> + ".file_harvest fh on if.id = fh.file_id"
      + "   where fh.last_change_run = (select last_run_no from " + <%=dbschema%> + ".index_job where name = ? )"
      + "   and fh.harvest_type = ?"
      + " ) as changed"

This query stored in the iUpdateSpatialExtent_begin.javajet.

utas-raymondng commented 1 year ago

@evacougnon from the log the AUV data is having issue when last run @2020-03-06, after that no more log entries even the last update happens at 2022-02-16

2019-06-12 13:47:00;akGf5e;akGf5e;akGf5e;AUV;AUV_harvester;Default;6;Java Exception;iPostgresqlDbUpdate_4;liquibase.exception.MigrationFailedException:Migration failed for change set changelog.xml::change_type_wms_time_column::talend:

2019-06-12 13:47:02;40VlVU;40VlVU;40VlVU;AUV;AUV_harvester;Default;6;Java Exception;iPostgresqlDbUpdate_4;liquibase.exception.MigrationFailedException:Migration failed for change set changelog.xml::change_type_wms_time_column::talend:

2019-06-13 09:50:40;QMSEeG;QMSEeG;QMSEeG;AUV;AUV_harvester;Default;6;Java Exception;iPostgresqlDbUpdate_4;liquibase.exception.MigrationFailedException:Migration failed for change set changelog.xml::change_type_wms_time_column::talend:

2019-06-13 09:50:42;acHGt9;acHGt9;acHGt9;AUV;AUV_harvester;Default;6;Java Exception;iPostgresqlDbUpdate_4;liquibase.exception.MigrationFailedException:Migration failed for change set changelog.xml::change_type_wms_time_column::talend:

2020-03-06 12:39:28;c8NAUV;c8NAUV;c8NAUV;ARGO;Argo_harvester;Default;6;Java Exception;iUpdateIndex_1;java.lang.RuntimeException:liquibase.exception.LockException: Could not acquire change log lock. Currently locked by fe80:0:0:0:8f6:71ff:fec3:eaf9%ens3 (fe80:0:0:0:8f6:71ff:fec3:eaf9%ens3) since 3/5/20 11:18 PM;1

Image

I suspect the harvester failed or changed so that the code to create the bounding box is not call.

utas-raymondng commented 1 year ago

Ignore this theory of error as I found log entry to proof this is not causing problem, this task run successfully and yet the job name quote the same.

***** completed finally block ***********
In finally block
INFO 2/28/23 1:39 PM:liquibase: Successfully acquired change log lock
INFO 2/28/23 1:39 PM:liquibase: Reading from srs_oc_soop_rad.databasechangelog
INFO 2/28/23 1:39 PM:liquibase: Reading from srs_oc_soop_rad.databasechangelog
INFO 2/28/23 1:39 PM:liquibase: Successfully released change log lock
* iUpdateIndex: FileResource[basePath: /mnt/ebs/tmp/SrsOcSoopRadHandlerlvt2ic06/temp/talend_basebo6evs5l, relativePath: IMOS/SRS/OC/radiometer/VMQ9273_Solander/2022/fv02-products/IMOS_SRS-OC_F_20221030T033446Z_VMQ9273_FV02_DALEC_END-20221030T073718Z.nc] is new or has been modified
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
* iNewOrModifiedFileList_3 modified/new resource file_id=998
* iNewOrModifiedFileList_1 modified/new resource file_id=998
* iUpdateSpatialExtent - files changed: true
* iUpdateSpatialExtent - updating spatial extent: true
* iUpdateSpatialExtent - updating uuid '28f8bfed-ca6a-472a-84e4-42563ce4df3f' from srs_oc_soop_rad.srs_oc_soop_rad_trajectory_map.geom using 3 degree resolution
log4j:WARN No appenders could be found for logger (org.apache.commons.httpclient.HttpClient).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

--- END TALEND OUTPUT ---
2023-02-28 13:39:15,122 INFO tasks.SRS_OC_SOOP_RAD[be98b87b-2d86-4c17-b475-026e66037bcd] updated file 'IMOS_SRS-OC_F_20221030T033446Z_VMQ9273_FV02_DALEC_END-20221030T073718Z_C-20230226T224944Z.nc': {'is_stored': True}
2023-02-28 13:39:15,123 SYSINFO tasks.SRS_OC_SOOP_RAD[be98b87b-2d86-4c17-b475-026e66037bcd] SrsOcSoopRadHandler transitioned to state: HANDLER_PUBLISHED
2023-02-28 13:39:15,123 SYSINFO tasks.SRS_OC_SOOP_RAD[be98b87b-2d86-4c17-b475-026e66037bcd] `postprocess` not overridden by child class, skipping step
2023-02-28 13:39:15,123 SYSINFO tasks.SRS_OC_SOOP_RAD[be98b87b-2d86-4c17-b475-026e66037bcd] SrsOcSoopRadHandler transitioned to state: HANDLER_POSTPROCESSED
2023-02-28 13:39:15,124 SYSINFO tasks.SRS_OC_SOOP_RAD[be98b87b-2d86-4c17-b475-026e66037bcd] get_notify_runner -> NotifyRunnerAdapter()
2023-02-28 13:39:15,124 SYSINFO tasks.SRS_OC_SOOP_RAD[be98b87b-2d86-4c17-b475-026e66037bcd] SrsOcSoopRadHandler transitioned to state: HANDLER_NOTIFIED_SUCCESS
2023-02-28 13:39:15,124 INFO tasks.SRS_OC_SOOP_RAD[be98b87b-2d86-4c17-b475-026e66037bcd] handler result for input_file '/mnt/ebs/processing/SRS_OC_SOOP_RAD/be98b87b-2d86-4c17-b475-026e66037bcd/IMOS_SRS-OC_F_20221030T033446Z_VMQ9273_FV02_DALEC_END-20221030T073718Z_C-20230226T224944Z.nc': SUCCESS
2023-02-28 13:39:15,124 SYSINFO tasks.SRS_OC_SOOP_RAD[be98b87b-2d86-4c17-b475-026e66037bcd] SrsOcSoopRadHandler transitioned to state: HANDLER_COMPLETED_SUCCESS
2023-02-28 13:39:15,132 SYSINFO tasks.SRS_OC_SOOP_RAD[be98b87b-2d86-4c17-b475-026e66037bcd] IncomingFileStateManager.state -> 'FILE_SUCCESS'

Theory of JOBNAME and TYPE being quote with ""

For this dataset https://portal-systest.aodn.org.au/search?uuid=78d588ed-79dd-47e2-b806-d39025194e7e

The log file shows the following lines

2022-08-15 12:30:52,867 SYSINFO tasks.SRS_ALTIMETRY[1caf673d-6f7d-4c1a-ad32-d47c9be349a4] executing /mnt/ebs/talend/jobs/srs_altimetry-srs_altimetry/java/SRS_ALTIMETRY_harvester/SRS_ALTIMETRY_harvester_run.sh --context_param paramFile="/mnt/ebs/talend/jobs/srs_altimetry-srs_altimetry/etc/srs_altimetry-srs_altimetry.conf" --context_param base=/mnt/ebs/tmp/SrsAltHandlerj39vqsw_/temp/talend_base1gff4i1i --context_param fileList=/mnt/ebs/tmp/SrsAltHandlerj39vqsw_/temp/talend_base1gff4i1i/file_listxn5f19as.txt --context_param logDir=/mnt/ebs/log/pipeline/harvest
2022-08-15 12:30:52,867 INFO tasks.SRS_ALTIMETRY[1caf673d-6f7d-4c1a-ad32-d47c9be349a4] --- START TALEND OUTPUT ---
updated file 'IMOS_SRSALT_TCPS_20011217T015051Z_BASJAS_FV01_SBE37d11m.nc': {'is_harvested': True}
INFO 8/15/22 12:30 PM:liquibase: Successfully acquired change log lock
INFO 8/15/22 12:30 PM:liquibase: Reading from srs_altimetry.databasechangelog
INFO 8/15/22 12:30 PM:liquibase: Reading from srs_altimetry.databasechangelog
INFO 8/15/22 12:30 PM:liquibase: Successfully released change log lock
* iUpdateIndex: FileResource[basePath: /mnt/ebs/tmp/SrsAltHandlerj39vqsw_/temp/talend_base1gff4i1i, relativePath: IMOS/SRS/ALTIMETRY/calibration_validation/SRSBASJAS/CTD_timeseries/IMOS_SRSALT_TCPS_20011217T015051Z_BASJAS_FV01_SBE37d11m.nc] is new or has been modified
Start of try block
tempFile_cid /tmp/talend-temp-8986280040131185496/changelog.xml
***********************
conn host:dbprod.emii.org.au, port:5432, schema:srs_altimetry, name:harvest?loginTimeout=1000&ssl=true&sslfactory=org.postgresql.ssl.NonValidatingFactory, user:srs_altimetry
url_cid:jdbc:postgresql://dbprod.emii.org.au:5432/harvest?loginTimeout=1000&ssl=true&sslfactory=org.postgresql.ssl.NonValidatingFactory
schema_cid:srs_altimetry
userName_cid:srs_altimetry
INFO 8/15/22 12:30 PM:liquibase: Successfully acquired change log lock
INFO 8/15/22 12:30 PM:liquibase: Reading from srs_altimetry.databasechangelog
INFO 8/15/22 12:30 PM:liquibase: Reading from srs_altimetry.databasechangelog
INFO 8/15/22 12:30 PM:liquibase: Successfully released change log lock
***** completed finally block ***********
In finally block
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
* iNewOrModifiedFileList_1 modified/new resource file_id=353
* iUpdateSpatialExtent - files changed: false
* iUpdateSpatialExtent - updating spatial extent: false

--- END TALEND OUTPUT ---
2022-08-15 12:31:11,947 INFO tasks.SRS_ALTIMETRY[1caf673d-6f7d-4c1a-ad32-d47c9be349a4] updated file 'IMOS_SRSALT_TCPS_20011217T015051Z_BASJAS_FV01_SBE37d11m.nc': {'is_stored': True}

tasks.SRS_ALTIMETRY.log

The query that create the change / no change for is show below, I execute it in srs_altimetry schema of harvest db


select exists (
select 1
from indexed_file if
         left join file_harvest fh on if.id = fh.file_id
          where fh.last_change_run = (select last_run_no from index_job where name = 'index' )
          and fh.harvest_type = 'data'
) as changed;
`

However it yield "true" when run it directly using sql console to read-only db of harvester, this means that it should execute the spatial extent code, so the error can either be the name or harvest_type argument during pipeline execution

I noticed that the two value is &quot; in harvester setup, which mean its value is a string with ""

    <elementParameter field="TEXT" name="JOBNAME" value="&quot;index&quot;"/>
    <elementParameter field="TEXT" name="HARVEST_TYPE" value="&quot;data&quot;"/>
utas-raymondng commented 1 year ago

Did a workaround and create a utility tools to update the mini-map in geonetwork directly, code and how to use it in PR

utas-raymondng commented 1 year ago

@evacougnon is happy about the change, @vietnguyengit feel free to Dod it

vietnguyengit commented 1 year ago

will do

vietnguyengit commented 1 year ago

portal systest still offline

vietnguyengit commented 1 year ago

I think this kind of utility tool, given its nature related to generating information, should have Unit tests to guarantee data integrity/correctness if there are future updates to maintain the tool.

vietnguyengit commented 1 year ago

https://www.notion.so/imos-world/Definition-of-Done-DoD-b98281f003a44ce890dfba5af2f00337

I'll move this back to in progress for test-related criteria in DoD definition, need tests unless there are specific reasons to not have tests.

utas-raymondng commented 1 year ago

@vietnguyengit the testing should be very low priority, I doubt if it make sense to investigate time to do unit testing

vietnguyengit commented 1 year ago

@vietnguyengit the testing should be very low priority, I doubt if it make sense to investigate time to do unit testing

I will move this ticket to DoD because the effort to build unit tests seem to be unnecessary.

vietnguyengit commented 1 year ago

no changes I added to what @utas-raymondng did, which were satisfying so I DoDed this ticket.