Closed lbesnard closed 9 years ago
for reference, @anguss00 dropped the schema on dbprod adn deployed the latest version of the harvester. An hour after starting it from scratch, the liquibase statement is still stuck at the beginning, and the schema is completely empty
@lbesnard, liquibase is also used by the indexing components to manage their own supporting tables. That appears to be what is creating the logging above. However, the liquibase changes themselves are happening very quickly, there is no time difference between liquibase grabbing a lock to do its updates and releasing the lock. Its more likely that the time is being spent iterating through all the files to determine what has changed, and from the above it looks like the first indexing component is the one that is taking most of the time.
@jonescc Thanks for looking at this. There is a problem in the indexing then.
All the files under /mnt/imos-t4/IMOS/public/AUV/auv_viewer_data/thumbnails are in context.exclude , which is set up in the 3 iIndexFiles components. This folder contains more than 6millions of files.
I suspected at the beginning that the indexing was the problem, and tried then to change the 3 different context.include to force which files to harvest. Either there is a problem with my regexp, either the component doesn't behave as it should.
any ideas ?
The AUV VIEWER TRACK harvester is extremely slow to go through the liquibase statements, even though there is no Liquibase changes.
On NSP14, for example the output of cat /mnt/ebs/talend/jobs/auv_viewer_track_rc-auv_viewer_track_rc/log/console.2014-11-23-19-20-41.log
shows that more than 3 hours is needed to do nothing ... and sometimes it can take 6 hours.
Some investigation was done with @anguss00 and @julian1 on dbrc, and the process seems to be idle.
It's worth mentioning this takes 30secondes on my local machine.
Here is the content of the liquibase statements.