search

bro-n-bro / cybernode

🧠 Provider for the Great Web.
https://cybernode.ai
Other
30 stars 11 forks source link

Execute up.sh failed #40

Closed MicrowaveDev closed 6 years ago

MicrowaveDev commented 6 years ago

Error on executing up.search.api.sh and up.ethereum.index.sh parts of up.sh script.

+++ dirname ./up.browser.sh
++ cd .
++ pwd
+ DIR=/Users/jonybang/workspace/cybernode
+ export COMPOSE_FILE=/Users/jonybang/workspace/cybernode/browser/docker-compose.yml
+ COMPOSE_FILE=/Users/jonybang/workspace/cybernode/browser/docker-compose.yml
+ docker-compose pull
Pulling cyber-ui (cybernode/cui-browser:latest)...
latest: Pulling from cybernode/cui-browser
Digest: sha256:ba50623031eeb41f6de705b50d403915330f2a03ec9dbf49641ad65a4a936d76
Status: Image is up to date for cybernode/cui-browser:latest
+ docker-compose up -d
Creating browser_cyber-ui_1 ... done
+ docker system prune -f
Total reclaimed space: 0B
+++ dirname ./up.chaingear-api.sh
++ cd .
++ pwd
+ DIR=/Users/jonybang/workspace/cybernode
+ export COMPOSE_FILE=/Users/jonybang/workspace/cybernode/chaingear-api/docker-compose.yml
+ COMPOSE_FILE=/Users/jonybang/workspace/cybernode/chaingear-api/docker-compose.yml
+ docker-compose pull
Pulling chaingear-api (cybernode/chaingear-api:latest)...
latest: Pulling from cybernode/chaingear-api
f2b6b4884fc8: Already exists
4fb899b4df21: Pull complete
74eaa8be7221: Pull complete
2d6e98fe4040: Pull complete
452c06dec5fa: Pull complete
7b3c215894de: Pull complete
c62e8e1f22ef: Pull complete
93263a0b6217: Pull complete
40b3e83d198c: Pull complete
525fb9449e34: Pull complete
6433e90e07bc: Pull complete
58f067208a0a: Pull complete
941c0de3f657: Pull complete
Digest: sha256:e79895504f7d81eeafd4e275710661b5c3564b74685d8045c54ae6a3df7d882c
Status: Downloaded newer image for cybernode/chaingear-api:latest
+ docker-compose up -d
Creating chaingearapi_chaingear-api_1 ... done
+ docker system prune
+ -yes
./up.chaingear-api.sh: line 10: -yes: command not found
+ CYBERNODE_REPOSITORY=/cyberdata/cybernode
+ SEARCH_REPOSITORY=/cyberdata/repositories/cyber-search
+ cd /cyberdata/repositories/cyber-search
./up.ethereum.index.sh: line 8: cd: /cyberdata/repositories/cyber-search: No such file or directory
+ git reset --hard
HEAD is now at aff86cf add configuration for deploy exchanges connector with metrics to gcloud
+ git pull
Already up to date.
+ cd /cyberdata/cybernode
./up.ethereum.index.sh: line 11: cd: /cyberdata/cybernode: No such file or directory
+ export COMPOSE_FILE=/cyberdata/cybernode/index/ethereum.yml
+ COMPOSE_FILE=/cyberdata/cybernode/index/ethereum.yml
+ docker-compose stop pump-eth
ERROR: .FileNotFoundError: [Errno 2] No such file or directory: '/cyberdata/cybernode/index/ethereum.yml'
+ docker build -t local-build/pump-ethereum -f /cyberdata/repositories/cyber-search/pumps/ethereum/Dockerfile /cyberdata/repositories/cyber-search
unable to prepare context: path "/cyberdata/repositories/cyber-search" not found
+ docker-compose stop dump-eth
ERROR: .FileNotFoundError: [Errno 2] No such file or directory: '/cyberdata/cybernode/index/ethereum.yml'
+ docker build -t local-build/dump-ethereum -f /cyberdata/repositories/cyber-search/dumps/ethereum/Dockerfile /cyberdata/repositories/cyber-search
unable to prepare context: path "/cyberdata/repositories/cyber-search" not found
+ docker-compose stop address-summary-eth
ERROR: .FileNotFoundError: [Errno 2] No such file or directory: '/cyberdata/cybernode/index/ethereum.yml'
+ docker build -t local-build/address-summary-eth -f /cyberdata/repositories/cyber-search/address-summary/ethereum/Dockerfile /cyberdata/repositories/cyber-search
unable to prepare context: path "/cyberdata/repositories/cyber-search" not found
+ docker-compose up -d
ERROR: .FileNotFoundError: [Errno 2] No such file or directory: '/cyberdata/cybernode/index/ethereum.yml'
+ docker system prune -f
Total reclaimed space: 0B
+ CYBERNODE_REPOSITORY=/cyberdata/cybernode
+ SEARCH_REPOSITORY=/cyberdata/repositories/cyber-search
+ cd /cyberdata/repositories/cyber-search
./up.search.api.sh: line 8: cd: /cyberdata/repositories/cyber-search: No such file or directory
+ git reset --hard
HEAD is now at aff86cf add configuration for deploy exchanges connector with metrics to gcloud
+ git pull
Already up to date.
+ cd /cyberdata/cybernode
./up.search.api.sh: line 11: cd: /cyberdata/cybernode: No such file or directory
+ export COMPOSE_FILE=/cyberdata/cybernode/index/search-api.yml
+ COMPOSE_FILE=/cyberdata/cybernode/index/search-api.yml
+ docker-compose stop
ERROR: .FileNotFoundError: [Errno 2] No such file or directory: '/cyberdata/cybernode/index/search-api.yml'
+ docker build -t local-build/search-api -f /cyberdata/repositories/cyber-search/search-api/Dockerfile /cyberdata/repositories/cyber-search
unable to prepare context: path "/cyberdata/repositories/cyber-search" not found
+ docker-compose up -d
ERROR: .FileNotFoundError: [Errno 2] No such file or directory: '/cyberdata/cybernode/index/search-api.yml'
+ docker system prune -f
Total reclaimed space: 0B
hleb-albau commented 6 years ago

Hi @Jonybang, current sh scripts are used for our staging servers and will be deleted soon. We will run all services atop of kubernetes. Right now I am working on user cli interface #24. If you still need to run this scripts, your should clone our search repo into /cyberdata/repositories/cyber-search BTW, could you, please, describe your case? Also, we have public alpha endpoints:

MicrowaveDev commented 6 years ago

My case: use your project with api endpoints on my custom network based on ethereum. I need to run your project with custom parity configuration and get my network data through your api endpoints.

hleb-albau commented 6 years ago

@Jonybang Okey, it is possible. I will give you example of docker-compose file for you case in few minutes. Do you need monitoring?

MicrowaveDev commented 6 years ago

Most likely it will come in handy in the future, but for a start i just need a search api for my custom network

hleb-albau commented 6 years ago

@Jonybang Here you can find our components overview. Here you can find docker images of this components. I prepared docker-compose file for you.

version: '3.3'
services:

  zoo:
    container_name: zoo
    image: confluentinc/cp-zookeeper:4.0.0-3
    environment:
      ZOOKEEPER_TICK_TIME: 2000
      ZOOKEEPER_CLIENT_PORT: 2181
      KAFKA_LOG4J_ROOT_LOGLEVEL: ERROR
    volumes:
      - /cyberdata/staging/zoo/data:/var/lib/zookeeper/data
      - /cyberdata/staging/zoo/log:/var/lib/zookeeper/log
    ports:
      - "2181:2181"

  kafka:
    container_name: kafka
    image: confluentinc/cp-kafka:4.0.0-3
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zoo:2181
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
      KAFKA_JMX_PORT: 1099
      KAFKA_LOG4J_ROOT_LOGLEVEL: ERROR
      KAFKA_LOG4J_ROOT_LOGLEVEL: WARN
      KAFKA_DELETE_TOPIC_ENABLE: "true"
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
      KAFKA_DEFAULT_REPLICATION_FACTOR: 1
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
    volumes:
      - /cyberdata/staging/kafka:/var/lib/kafka
      - /cyberdata/staging/kafka/data:/var/lib/kafka/data
    ports:
      - "9092:9092"
      - "1099:1099"

  kafka-manager:
    container_name: kafka-manager
    image: hlebalbau/kafka-manager:latest
    environment:
      ZK_HOSTS: "zoo:2181"
      APPLICATION_SECRET: "random-secret"
    command: -Dpidfile.path=/dev/null
    ports:
      - "9000:9000"

  elassandra-search:
    container_name: elassandra-search
    image: strapdata/elassandra:5.5.0.10
    volumes:
      - /cyberdata/staging/elassandra-search/lib:/var/lib/cassandra
      - /cyberdata/staging/elassandra-search/log:/var/log/cassandra
    environment:
      CASSANDRA_DC: DEVELOPMENT
      CASSANDRA_CLUSTER_NAME: CYBER_SEARCH
    ports:
      - "9042:9042"
      - "9200:9200"
      - "9300:9300"

  pump-eth:
    container_name: pump-eth
    image: cybernode/chain-pump-ethereum:0.2.4
    restart: unless-stopped
    environment:
      KAFKA_BROKERS: kafka:9092
      KAFKA_TRANSACTION_BATCH: 1000
      CHAIN: ETHEREUM
      CHAIN_NODE_URL: http://parity:8545

  dump-eth:
    container_name: dump-eth
    image: cybernode/chain-dump-cassandra-ethereum:0.2.4
    restart: unless-stopped
    environment:
      CASSANDRA_HOSTS: elassandra-search
      KAFKA_BROKERS: kafka:9092
      CHAIN: ETHEREUM

  contract-summary-eth:
    container_name: address-summary-eth
    image: cybernode/chain-contract-summary-ethereum:0.2.4
    restart: unless-stopped
    environment:
      CASSANDRA_HOSTS: elassandra-search
      KAFKA_BROKERS: kafka:9092
      CHAIN: ETHEREUM

  search-api:
    container_name: search-api
    image: cybernode/search-api
    restart: unless-stopped
    environment:
      CASSANDRA_HOSTS: elassandra-search
    ports:
      - "8080:8080"
  1. Replace /cyberdata/staging for you desired host folder. Or, you can delete it at all, if you want just to play with cybernode and store data in container directly.
  2. Replace http://parity:8545 with you node URL. Note, do not delete prefix "http"
  3. Then populate cassandra with keyspaces using this guide

Than, check that all components are running and no error occurs by

docker logs pump-eth --tail 100
docker logs dump-eth --tail 100
docker logs contract-summary-eth --tail 100

If you have any troubles just chat me, or we can make a call.

MicrowaveDev commented 6 years ago

if you have a telegram account for chat with me - it would be convenient

hleb-albau commented 6 years ago

@Jonybang https://t.me/hleb_albau