Xchange crawler setup

[abitrolly]

Xchange crawler from https://github.com/cyberFund/xchange-crawler is only mentioned in architecture diagram without any links to further info or setup instructions.

UPDATE: I found the sources.

[abitrolly]

Architecture diagram needs a better 3D art to make the diagram interactive and clickable:

[abitrolly]

Quick and dirty way to make it running

docker run -d -u $(id -u cyber) --name rethink -p 127.0.0.1:8080:8080 -p 127.0.0.1:28015:28015 -v /home/cyber/cyberdata/rethink:/data rethinkdb:2.3
cd images/xchange-crawler
./build-image.sh

Find out interface rethink is listening:

docker logs rethink

Add that interface address port to custom crawler.properties and run crawler:

docker run -d -u $(id -u cyber) --name crawler -v $PWD/crawler.properties:/cyberapp/crawler.properties xchange-crawler

Cleanup

• [ ] document rethink ports
• [x] find a place for configuring rethink startup and xchange crawler link
• [ ] find rethink container interface automatically by container name
• [ ] add run infrastructure to run and down crawler in single steps

[abitrolly]

RethinkDB 2.3 moved to new authentication scheme (from authkey to password). The solution for now is to run crawler user unauthenticated.

Access dashboard by forwarding listening local port to remote service:

ssh -v -NL 127.0.0.1:8080:127.0.0.1:8080 mars.cybernode.cyber.fund

Adding user through RethinkDB admin interface:

r.db('rethinkdb').table('users').insert({id: 'crawler', password: false})

Configuring crawler.properties:

rethink.authKey=

[abitrolly]

RethinkDB is reporting issues:

The following server encountered a memory problem:

135ddb5698a9_rac The issue reported is: Some RethinkDB data on this server has been placed into swap memory in the past hour. This may impact performance.

Please fix the problem that is causing the server to use swap memory. This issue will go away after an hour has passed since swap memory was used, or after you restart RethinkDB.

[abitrolly]

Looks like xchange-crawler is too memory hungry.

21184 root      20   0 5029972 3,414g  25156 S 188,4  5,4  80:00.34 rethinkdb
25573 cyber     20   0 18,424g 7,210g  15120 S  58,5 11,5  29:14.86 java
12294 cyber     20   0  734020 593616  10340 S   1,0  0,9  23046:09 btcd

[abitrolly]

xhange-crawler is killed and replaced by cyber-markets (#79).