docker-compose / cache / queues

[lucmski]

Hi,

Hope you are all well !

Just was wondering if hoaxy would be soon released as a docker-compose for the front and back end as a quick starter.

Have u added httpcache (x-vary) middleware (eg https://github.com/gregjones/httpcache) or twitter accounts queueing to the task list (eg. https://github.com/Nedja995/twint_server) ?

Thanks for the hard work ^^

Cheers

[filmenczer]

Working on Docker (#26). In the meanwhile, an AMI is available for EC2 (see https://github.com/IUNetSci/hoaxy-backend#hoaxy-backend-as-aws-image-ami).

Not sure about other questions...

[lucmski]

I meant is it possible to distribute a list of twitter usernames to fetch and get their botometer score async like with payloads managed by rabbitmq or celery. Globally, it is more about handling more data and being able to resume tasks. (get the last tweet id scraped so it prevents using resources).

[lucmski]

Example for a front/server dockerfile (alpine/nodejs v11.x): https://gist.github.com/lucmski/b4a143b0bf026448f159e6bf463ed0c2

Example for hoaxy-backend (alpine 3.9/mininconda2): https://gist.github.com/lucmski/87f13b669422c2f057b500404a01d359

If needed, I can do a miniconda3...

[chathuriw]

Thank you for your contribution @lucmski. I will check the docker files and let you know.

[chathuriw]

@lucmski, I'm trying the docker file for hoaxy-backend. I was able to follow first three steps. I'm stucked with step 4 (https://gist.github.com/lucmski/87f13b669422c2f057b500404a01d359#run-docker-container). I think there is a typo in the first command in that step. (docker -ti --rm lucmichalskip/hoaxy-backend:alpine-miniconda2-jdk8 hoaxy -h)

I'm getting this error. "unknown shorthand flag: 't' in -ti"

[lucmski]

forgot to add 'run' command sorry, the gist was updated

[chathuriw]

@lucmski Can you share the Dockerfile lucmichalskip/hoaxy-backend:alpine-miniconda2-jdk8 with us. I'm trying to follow your instructions again and got an error in docker run command. I think it will be easier to troubleshoot if I can see the Dockerfile.

chathuri@chathuri-Latitude-E7450:~/IUNI/hoaxy-backend$ docker run -ti --rm lucmichalskip/hoaxy-backend:alpine-miniconda2-jdk8 hoaxy -h /opt/conda/lib/python2.7/site-packages/hoaxy-0.1.0-py2.7.egg/hoaxy/init.py:30: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details. Traceback (most recent call last): File "/opt/conda/bin/hoaxy", line 11, in load_entry_point('hoaxy==0.1.0', 'console_scripts', 'hoaxy')() File "build/bdist.linux-x86_64/egg/hoaxy/commands/cmdline.py", line 80, in main File "build/bdist.linux-x86_64/egg/hoaxy/utils/init.py", line 20, in list_cls_under_mod File "/opt/conda/lib/python2.7/importlib/init.py", line 37, in import_module import(name) File "build/bdist.linux-x86_64/egg/hoaxy/commands/crawl.py", line 12, in File "build/bdist.linux-x86_64/egg/hoaxy/crawl/spiders/article.py", line 11, in File "build/bdist.linux-x86_64/egg/hoaxy/database/init.py", line 27, in File "/opt/conda/lib/python2.7/site-packages/sqlalchemy/engine/init.py", line 423, in create_engine return strategy.create(*args, kwargs) File "/opt/conda/lib/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 87, in create dbapi = dialect_cls.dbapi(dbapi_args) File "/opt/conda/lib/python2.7/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py", line 605, in dbapi import psycopg2 File "/opt/conda/lib/python2.7/site-packages/psycopg2/init.py", line 50, in from psycopg2._psycopg import BINARY, NUMBER, STRING, DATETIME, ROWID ImportError: libssl.so.1.0.0: cannot open shared object file: No such file or directory

[lucmski]

Hi,

Sounds that even my dockerfile is outdated; i need to update it. Why do not you split into separate service with a docker-compose ?

My telegram is @deepocrates for faster communication.

Cheers, L

[roscopecoltran]

From fork, https://github.com/lucmski/hoaxy-backend/tree/dockerize-p3/docker

please try:

git clone https://github.com/lucmski/hoaxy-backend
cd hoaxy-backend/docker
docker build -t hoaxy .
docker run -ti --rm hoaxy -h

[lucmski]

should be interesting you https://github.com/twintproject/twint-search

[chathuriw]

@roscopecoltran, @deepocrates I was able to build docker for hoaxy backend successfully with your instructions. Thank you for the contribution.

But when I run hoaxy -h, it gives me this error.

(base) chathuri@chathuri-Latitude-E7450:~/IUNI/hoaxy/lucmski/hoaxy-backend/docker$ docker run -ti --rm hoaxy -h /opt/conda/lib/python3.7/site-packages/psycopg2-2.7.6.1-py3.7-linux-x86_64.egg/psycopg2/init.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: http://initd.org/psycopg/docs/install.html#binary-install-from-pypi. """)

**** ERROR ** NO HOAXY CONFIGURATION FILE FOUND: (DEFAULT '/home/.hoaxy/conf.yaml')!

(1) If you have put the conf.yaml to somewhere other than default location '/root/.hoaxy/', please set your enviroment variable HOAXY_HOME properly. (2) Use hoaxy config [--home=] to get a sample of configuration file named conf.sample.yaml (3) Edit and rename conf.sample.yaml to conf.yaml. And you should edit and rename other sample files.

Currently, hoaxy uses default settings and may not work properly.

---

Usage: hoaxy [options] [...] hoaxy -h | --help hoaxy -v | --version

Subcommands are: config Generate sample of configuration files crawl Crawl news sites init Initialize DB tables and load site data if exists site News site management: add, alter and dump lucene Lucene Indexing and Searching report Report and generate statistics of hoaxy collection sns Online social network services management

For subcommands, use hoaxy <command> --help to get the usage of the subcommand !

Global options are: --help Show help for this subcommand. --console-log-level= Set logging level for console output. [default: debug]

[chathuriw]

@lucmski merged the docker changes to master. Thank you for your contribution.