Download results with CLI

[MlataIbrahim]

Hello Crawlab team, i'm using crawlab to deploy my scrapy spiders and when I'm trying to download results CSV it takes a lot of time sometimes more than 15min , is there any commands line with CLI sdk to download directly the data . thank you

[tikazyq]

Hi @MlataIbrahim ,

The results are stored in MongoDB. If your result dataset is very large, downloading directly from the web could be problematic. The best way is to export from MongoDB using mongoexport CLI. Perhaps you can refer to the documentation below.

Docs for mongoexport: https://docs.mongodb.com/manual/reference/program/mongoexport/

Hope this would help.

[MlataIbrahim]

Thanks, @tikazyq for your response, I'm still struggling with connecting MongoDB database from my machine I used the host IP and port default and still get an error .

[tikazyq]

Thanks, @tikazyq for your response, I'm still struggling with connecting MongoDB database from my machine I used the host IP and port default and still get an error .

How do you connect to MongoDB? And how does your docker-compose.yml look like?

I guess you might have forgot to expose your MongoDB's port to the host machine.

[MlataIbrahim]

i'm trying to connect the server with MongoDB Compass ,my docker-compose.yml is as default

[tikazyq]

I think you need to change the mongo part in docker-compose.yml as below. And 27017 is the port to connect.

...
  mongo:
    image: mongo:latest
    restart: always
    # environment:
    #   MONGO_INITDB_ROOT_USERNAME: username
    #   MONGO_INITDB_ROOT_PASSWORD: password
    # volumes:
    #   - "/opt/crawlab/mongo/data/db:/data/db"  # make data persistent 持久化
    ports:
      - "27017:27017"  # expose port to host machine 暴露接口到宿主机
...

[MlataIbrahim]

i changed the docker-compose.yml file and tried to connect with compass mongodb://username:password@host_ip:27017/crawlab_test still get connection timed out

[tikazyq]

I think you need to raise the firewall for that port 27017

[MlataIbrahim]

i did this one also and get this error connect ECONNREFUSED host_ip:27017, i think there some issue with Docker network