Norconex / committer-sql

Implementation of Norconex Committer for SQL (JDBC) databases.
https://opensource.norconex.com/committers/sql/
Apache License 2.0
1 stars 6 forks source link

I got an issue in mapping the fields. #17

Open haolin96 opened 9 months ago

haolin96 commented 9 months ago

Hello,

I have configured the SQLFileCommitter.xml and the content is `<committer class = "com.norconex.committer.sql.SQLCommitter">

`

And I have set the SQLCommitterConfig by using the methods:

`SQLCommitterConfig sqlCommitterConfig = new SQLCommitterConfig();

    sqlCommitterConfig.setDriverClass("com.mysql.cj.jdbc.Driver");
    sqlCommitterConfig.setConnectionUrl("jdbc:mysql://localhost:3306/sys");
    sqlCommitterConfig.setCredentials(new Credentials().setUsername("root").setPassword("123456"));
    sqlCommitterConfig.setTableName("test");
    sqlCommitterConfig.setPrimaryKey("id");
    sqlCommitterConfig.setFixFieldNames(true);
    sqlCommitterConfig.setFixFieldValues(true);
    sqlCommitterConfig.setTargetContentField("content");

    XML xml = new XML(Path.of("fraud\\src\\main\\resources\\SQLFileCommitter.xml"));
    SQLCommitter sqlCommitter = new SQLCommitter(sqlCommitterConfig);
    sqlCommitter.loadFromXML(xml);
    sqlCommitter.loadCommitterFromXML(xml);
    crawlerConfig.setCommitters(sqlCommitter);`

I want to store only part of the extracted fields into MYSQL. But I got the exception Caused by: java.sql.SQLException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '-Backend-Response,Server,x-webkit-csp,Content-Location,x-frame-options,x-cdn-pro' at line 1 Query: INSERT INTO test(X-Backend-Response,Server,x-webkit-csp,Content-Location,x-frame-options,x-cdn-provider,Referrer-Policy,X-SecNG-Response,dc:title,Content-Encoding,Set-Cookie,collector.depth,id,surrogate-control,document_reference,google-site-verification,document.contentEncoding,strict-transport-security,pragma,x-xss-protection,x-idc-id,Cache-Control,document.contentType,Content-Language,expires,document.contentFamily,renderer,collector.redirect-trail,force-rendering,description,title,content,x-edge-timing,x-content-security-policy,X-Cache-Lookup,collector.is-crawl-new,collector.http-fetcher,Content-Length,Content-Type,Transfer-Encoding,X-Parsed-By,Connection,Date,X-UA-Compatible,content-security-policy,x-content-type-options,viewport,x-lb-timing,X-NWS-LOG-UUID,Vary) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) Parameters: [0.028, CLOUD ELB 1.0.0, default-src blob:; img-src data: blob: resource: t.captcha.qq.com .dun.163yun.com .dun.163.com .126.net .nosdn.127.net nos.netease.com;

It seemed like the insert query was too long and the fileds mapping didn't work, so I didn't have enough columns to store. How can I deal with that?

ohtwadi commented 9 months ago

If you only want to keep the document.reference metadata field, then I suggest you use KeepOnlyTagger.