platonai / exotic-amazon

A complete solution to crawl amazon at scale completely and accurately.
143 stars 46 forks source link

How do I change the default store mode #25

Open ws4435700 opened 1 year ago

ws4435700 commented 1 year ago

Hi, I've set up jdbcommitter as required, and commented out the code in the configuration to load mongodb by default, but the default mode remains once the service is started

platonai commented 1 year ago

Mongodb has to be started, it is used to store the metadata, and it's really simple to use.

JdbCommitter is used to save the extract result to a JDBC compatible database, and the metadata is still saved in MongoDB.

All metadata is saved to WebPage and persisted to MongoDB.

ws4435700 commented 1 year ago

Mongodb 必须启动,它是用来存储元数据的,而且使用起来非常简单。

JdbCommitter用于将提取结果保存到JDBC兼容的数据库中,元数据仍然保存在MongoDB中。

所有元数据都保存到 WebPage 并持久化到 MongoDB。

So how do I configure to use a jdbc database? I see that the relevant configuration is already in the code, but it has not been called, and there is no corresponding sql to write to the database. Do I need to write this code now, or just add the configuration to write to the jdbc database

platonai commented 1 year ago

JDBC config file config/jdbc-sink-config.json has been deprecated, you should set up JdbCommitter programmatically.

You can use schema-dump.sql to create your MySQL tables.

The config file extract-config.json is a mapping from the extract rules written in X-SQL to the destination tables in database.