search

Norconex / committer-sql

Implementation of Norconex Committer for SQL (JDBC) databases.
https://opensource.norconex.com/committers/sql/
Apache License 2.0
1 stars 6 forks source link

SQLCommitter not storing data #14

Open Rahul9354 opened 1 year ago

Rahul9354 commented 1 year ago

I am trying to store data in mysql table using sqlcommitter but unable to do so cause I don't understand where to do the mapping for fields which will then get converted to mysql table columns. I am getting the required data in a json file when using a jsonfilecommitter. If you could please share an example of the config file and tags that I should be using it would be a great help.

Rahul9354 commented 1 year ago

hello @essiembre Can you help me out regarding my issue.

ohtwadi commented 1 year ago

You can use <fieldMappings> to map fields extracted by the Crawler to fields you would like to appear in your DB. This is well documented in our docs

For example, the snippet below will result in the crawler description field mapped to desc_mapped in your DB

<fieldMappings>
  <mapping
    fromField="description"
    toField="desc_mapped"/>
</fieldMappings>
ohtwadi commented 1 year ago

Additionally, if you only have one committer, you can rename the metadata fields via the Importer's RenameTagger

Rahul9354 commented 1 year ago

@ohtwadi i have already renamed the meta tags with renameTagger I am getting issue in table creation and storage of data. Can the table be automatically created or do I have to use createTableSQL tag to manually create the table and how the renamed meta fields are mapped to the table. And what is the use of tags like restrictTo,fieldMatcher,valueMatcher,queue,
properties specifically I don't get. If you can share a sample xml configuration file it would be a great help.

ohtwadi commented 1 year ago

Can the table be automatically created or do I have to use createTableSQL tag to manually create the table

As per the docs:

By default, this Committer will throw an exception when trying to insert values into non-existing database table or fields. It is recommended you make sure your database table exists and the document fields being sent to the committer match your database fields. Alternatively, you can provide the necessary SQLs to create a new table as well as new fields as needed using CreateTableSQL and CreateFieldSQL respectively

If you are having issues with table creation, you may create the table manually.

how the renamed meta fields are mapped to the table

Table column names will be the same as the field name in the Committer.

what is the use of tags like restrictTo,fieldMatcher,valueMatcher

See AbstractCommitter.

Queue

If you are just starting out, I suggest not tinkering with this just yet. It's very likely that the default options will work just fine for you. More details can be found here FSQueue