Norconex / collector-filesystem

Norconex Filesystem Collector is a flexible crawler for collecting, parsing, and manipulating data ranging from local hard drives to network locations into various data repositories such as search engines.
http://www.norconex.com/collectors/collector-filesystem/
21 stars 13 forks source link

SQL Commiter with SCRAM-based authentication fails #64

Open schudoku opened 2 years ago

schudoku commented 2 years ago

I am using norconex filesystem crawler 2.9.1 and want to commit to a postgre database 13.4. The database requires user authentication which is SCRAM-based. In the collector config I specified the following commiter:

  <committer class="com.norconex.committer.sql.SQLCommitter">
    <driverPath>C:\dev\databases\drivers\postgresql-42.3.5.jar</driverPath>
    <driverClass>org.postgresql.Driver</driverClass>
    <connectionUrl>jdbc:postgresql://localhost:64000/databasename</connectionUrl>
    <tableName>crawler.crawl_test</tableName>

    <createTableSQL>
      CREATE TABLE {tableName} (
          {primaryKey} VARCHAR(32672) NOT NULL,
          content CLOB,
          PRIMARY KEY ( {primaryKey} ),
          title   VARCHAR(256)
          author  VARCHAR(256)
      )
    </createTableSQL>

    <createFieldSQL>
      ALTER TABLE {tableName} ADD {fieldName} VARCHAR(5000)
    </createFieldSQL>

    <fixFieldValues>true</fixFieldValues>

    <credentials>
      <username> ---PLAIN-USERNAME-HERE--- </username>
      <password> ---PLAIN-PW-HERE--- </password-->
    </credentials>

  </committer>

And this is the stacktrace I get:

[..] INFO [AbstractCrawler] Crawler For ProSi Reports: Crawler finishing: committing documents. INFO [AbstractFileQueueCommitter] Committing 266 files INFO [SQLCommitter] Sending 100 commit operations to SQL database. INFO [SQLCommitter] Checking if table "crawler.crawl_1" exists... INFO [SQLCommitter] Table "crawler.crawl_1" does not exist. Attempting to create it... INFO [AbstractCrawler] Crawler For ProSi Reports: Crawler executed in 4 seconds. ERROR [JobSuite] Execution failed for job: Crawler For ProSi Reports com.norconex.committer.core.CommitterException: Could not commit batch to database. at com.norconex.committer.sql.SQLCommitter.commitBatch(SQLCommitter.java:480) at com.norconex.committer.core.AbstractBatchCommitter.commitAndCleanBatch(AbstractBatchCommitter.java:179) at com.norconex.committer.core.AbstractBatchCommitter.cacheOperationAndCommitIfReady(AbstractBatchCommitter.java:208) at com.norconex.committer.core.AbstractBatchCommitter.commitAddition(AbstractBatchCommitter.java:143) at com.norconex.committer.core.AbstractFileQueueCommitter.commit(AbstractFileQueueCommitter.java:222) at com.norconex.committer.sql.SQLCommitter.commit(SQLCommitter.java:425) at com.norconex.collector.core.crawler.AbstractCrawler.execute(AbstractCrawler.java:274) at com.norconex.collector.core.crawler.AbstractCrawler.doExecute(AbstractCrawler.java:228) at com.norconex.collector.core.crawler.AbstractCrawler.startExecution(AbstractCrawler.java:184) at com.norconex.jef4.job.AbstractResumableJob.execute(AbstractResumableJob.java:49) at com.norconex.jef4.suite.JobSuite.runJob(JobSuite.java:357) at com.norconex.jef4.suite.JobSuite.doExecute(JobSuite.java:294) at com.norconex.jef4.suite.JobSuite.execute(JobSuite.java:167) at com.norconex.collector.core.AbstractCollector.start(AbstractCollector.java:150) at com.norconex.collector.core.AbstractCollectorLauncher.launch(AbstractCollectorLauncher.java:95) at com.norconex.collector.fs.FilesystemCollector.main(FilesystemCollector.java:76) Caused by: java.sql.SQLException: Cannot create PoolableConnectionFactory (The server requested SCRAM-based authentication, but no password was provided.) at org.apache.commons.dbcp2.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:2294) at org.apache.commons.dbcp2.BasicDataSource.createDataSource(BasicDataSource.java:2039) at org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1533) at org.apache.commons.dbutils.AbstractQueryRunner.prepareConnection(AbstractQueryRunner.java:204) at org.apache.commons.dbutils.QueryRunner.update(QueryRunner.java:419) at com.norconex.committer.sql.SQLCommitter.ensureTable(SQLCommitter.java:607) at com.norconex.committer.sql.SQLCommitter.commitBatch(SQLCommitter.java:463) ... 15 more Caused by: org.postgresql.util.PSQLException: The server requested SCRAM-based authentication, but no password was provided. at org.postgresql.core.v3.ConnectionFactoryImpl.lambda$doAuthentication$4(ConnectionFactoryImpl.java:791) at org.postgresql.core.v3.AuthenticationPluginManager.withPassword(AuthenticationPluginManager.java:81) at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:788) at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:180) at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:235) at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49) at org.postgresql.jdbc.PgConnection.(PgConnection.java:223) at org.postgresql.Driver.makeConnection(Driver.java:402) at org.postgresql.Driver.connect(Driver.java:261) at org.apache.commons.dbcp2.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:39) at org.apache.commons.dbcp2.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:256) at org.apache.commons.dbcp2.BasicDataSource.validateConnectionFactory(BasicDataSource.java:2304) at org.apache.commons.dbcp2.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:2290) ... 21 more INFO [JobSuite] Running Crawler For ProSi Reports: END (Mon May 23 09:41:40 CEST 2022)

It says no password provided, but I did. Could you help me?

Regards!

essiembre commented 1 year ago

Not sure if you still have the issue, but it should work with PostgreSQL JDBC drivers v42.2+