Data Sources

[Vr1llon]

Is your feature request related to a problem? Please describe. Data insertion into relational database (Postgresql in my case) using data sources. I had to dig into crawlab sdk source code to figure out how and what the crawlab item pipeline does exactly, for example does it create table or not. Does it insert the _tid and _pid or not. And what table crawlab is looking for before insertion. And what defines the column names? I had to do this because there was no information regarding this topic in the data sources documentation.

Describe the solution you'd like Additional information to the documentation of the data sources.

Describe alternatives you've considered

• Option for table creation if it does not exist
• Option for enable/disable _tid insertion
• Option to rearrange data fields

Other suggestions

• Give spiders ability to create files/folders during runtime

Additional note Thank you for the awesome software 🙏🏻

[llmontoryxd]

I'm also try to use crawlab, but have no idea how to use it with my postgresql