search

alephdata / memorious

Lightweight web scraping toolkit for documents and structured data.
https://docs.alephdata.org/developers/memorious
MIT License
311 stars 59 forks source link

Cannot find crawlers in list #124

Closed ghost closed 3 years ago

ghost commented 4 years ago

Hi,

Hope you are all well !

I tried to create a crawler, available at https://github.com/x0rzkov/memorious/tree/fmc/fmc. And, it cannot display the crawler fmc, neither, in the ui or the list from the cli.

Can you help me to point out what I missed configured ?

Thanks in advance for any insights or inputs on this issue.

Cheers, X

ghost commented 4 years ago

Found the issue, you need to add MEMORIOUS_CONFIG_PATH env in the docker-compose.

Now, I have another issue. How to transfer crawled data to aleph ?

ghost commented 4 years ago

I have setup the following in the docker compose both for UI/workder service :

But I do not load data crawled by memorious to alpeh.

Can you help me to fix this last issue ? please.

sunu commented 4 years ago

Are you using the aleph_emit op to push data into Aleph?

ghost commented 4 years ago

Nope, I just though that setting up these 2 en variables gonna be enough. How to use aleph_emit ?