search

OpenWIS / openwis

http://openwis.github.io/openwis
GNU General Public License v3.0
11 stars 15 forks source link

3.16 - Rebuild Issue #413

Open yvesgoupil opened 1 year ago

yvesgoupil commented 1 year ago

There is a problem with Solr re-indexing (Admin portal > Metadata Service >Index). It works with stop-gap metadatas but in the case where GISCs metadata has been harvested previously and only one metadata contains an error, in the bandbox for example, the re-indexing crashes and the whole metadatas are not accessible anymore. The only solution I found to be able to have the metadata available again by making new harvestings is to delete the tables concerning the metadata in the database.

yvesgoupil commented 1 year ago
abrmh commented 1 year ago

Changes has been made and committed to the branch 3.16 to avoid the Solr to crash when rebuilding indexes.

  1. The Admin and User portal wars have to be deployed
  2. A file called wrongmetada.log is created and will contain all wrong metadata's

  3. A file log4.cfg should look like as text bellow:

    Default Logging Configuration

    log4j.rootLogger = OFF

GEONETWORK SETTINGS

log4j.logger.geonetwork = INFO, jeeves log4j.logger.geonetwork.search = WARN log4j.logger.geonetwork.editorexpandelement = WARN log4j.logger.geonetwork.editoraddelement = WARN log4j.logger.geonetwork.index = INFO log4j.logger.geonetwork.csw = WARN log4j.logger.geonetwork.mef = WARN log4j.logger.geonetwork.z3950server = WARN log4j.logger.geonetwork.z3950 = WARN log4j.logger.geonetwork.sru = INFO log4j.logger.geonetwork.sru.search = INFO log4j.logger.geonetwork.oai = INFO log4j.logger.geonetwork.datamanager = INFO

JEEVES SETTINGS

log4j.logger.jeeves = INFO, jeeves log4j.logger.jeeves.service = INFO log4j.logger.jeeves.dbms = WARN log4j.logger.jeeves.dbmspool = WARN log4j.logger.jeeves.xlinkprocessor = INFO

OPENWIS SETTINGS

log4j.logger.Login = INFO, jeeves log4j.logger.statistics = INFO, statistics log4j.logger.openwis = INFO, jeeves log4j.logger.openwis.metadataaligner = INFO, metadata log4j.logger.openwis.extract.productmetadata = INFO, metadata log4j.logger.openwis.wrong.productmetadata = INFO, wrongmetadata log4j.logger.openwis.harvester.executor = INFO, metadata

log4j.logger.openwis.service.provider = DEBUG, jeeves

log4j.logger.openwis.service.management = DEBUG, jeeves

SPRING

log4j.logger.org.springframework = WARN, jeeves

JZKIT SETTINGS

log4j.logger.com.k_int=WARN, jeeves log4j.logger.org.jzkit=WARN, jeeves log4j.logger.org.jzkit.a2j=WARN, jeeves log4j.logger.org.jzkit.search.impl.LRUCache = INFO, jeeves

JEEVES APPENDER

log4j.appender.jeeves =org.apache.log4j.DailyRollingFileAppender log4j.appender.jeeves.DatePattern='.'yyyy-MM-dd log4j.appender.jeeves.file = ${user.home}/logs/openwis.log log4j.appender.jeeves.layout = org.apache.log4j.PatternLayout log4j.appender.jeeves.layout.ConversionPattern=%d{ISO8601} %-5p [%c] - %m%n

METADATA APPENDER

log4j.appender.metadata =org.apache.log4j.DailyRollingFileAppender log4j.appender.metadata.DatePattern='.'yyyy-MM-dd log4j.appender.metadata.file = ${user.home}/logs/metadata.log log4j.appender.metadata.layout = org.apache.log4j.PatternLayout log4j.appender.metadata.layout.ConversionPattern=%d{ISO8601} %-5p [%c] - %m%n

WRONG METADATA APPENDER

log4j.appender.wrongmetadata =org.apache.log4j.DailyRollingFileAppender log4j.appender.wrongmetadata.DatePattern='.'yyyy-MM-dd log4j.appender.wrongmetadata.file = ${user.home}/logs/wrongmetadata.log log4j.appender.wrongmetadata.layout = org.apache.log4j.PatternLayout log4j.appender.wrongmetadata.layout.ConversionPattern=%d{ISO8601} %-5p [%c] - %m%n

STATS APPENDER

log4j.appender.statistics = org.apache.log4j.RollingFileAppender log4j.appender.statistics.maxBackupIndex= 1 log4j.appender.statistics.maxFileSize= 1024KB log4j.appender.statistics.file = ${user.home}/logs/stats.csv log4j.appender.statistics.append = false log4j.appender.statistics.layout = org.apache.log4j.PatternLayout log4j.appender.statistics.layout.ConversionPattern=%d{ISO8601};%m%n

CONSOLE SETTINGS

log4j.appender.console = org.apache.log4j.ConsoleAppender log4j.appender.console.layout = org.apache.log4j.PatternLayout log4j.appender.console.layout.ConversionPattern=%d{ISO8601} %-5p [%c] - %m%n

abrmh commented 1 year ago

Solr is indexing all metadata when doing harvesting or rebuild without crashing , it list all wrong metadata's in wrongmetadata.log file but it the incorrect MDs are not corrected in database so it is up to admin to correct them.

yvesgoupil commented 1 year ago

I've deployed the last commit https://github.com/OpenWIS/openwis/commit/a65f519661fbc412fb4b20ed89f8f108c7b20159 made by Brahim, I tested this fix and it works perfectly. Here is the usecase:

Stop data feed Stop user and admin portals Stop solr Delete MDs in database OpenWIS=# delete FROM metadata where istemplate = 'n' ; OpenWIS=# delete FROM openwis_pattern_metadata_mapping ; OpenWIS=# delete FROM openwis_product_metadata ; Launch solr Start user and admin portals Restart the data feed Harvest the MF set for example See that metadata with errors appear in the wrongmetadata.log. Check the number of metadata of the test set in the catalog on the admin portal. Do a Re-build on the admin portal Check that the same metadata with errors appear again in the log wrongmetadata.log. Check that the number of test set metadata in the catalog on the admin portal has not changed. There has been no Solr crash.

I propose that we consider this issue resolved.

yvesgoupil commented 1 year ago

I set the label to "close me".

Yves.