Got 403 for configuration service url

[saurabhPV]

I am using dev mode docker-compose -f docker-compose-with-elasticsearch.yaml up -d I can see nginx and all other containers are running other than arc service because it doesn't have env I suppose since it throws an error for the same but then 403

[lakhansamani]

@saurabhPV can you share the logs of docker container of nginx + arc Ref for logs command: https://docs.docker.com/engine/reference/commandline/logs/

Also do check if any other service is not running on port 80

[saurabhPV]

thanks @lakhansamani for quick response! I can see there is no process running on 80 when I stopped the containers - also it would not have start the container had the port been occupied.

these are the logs from arc service

these are the logs from nginx service

Also I tried to access the nginx access and error logs from \var\logs\nginx and I don't see any entry there either when I try to access http://localhost/

[lakhansamani]

@saurabhPV seems some issues with windows and docker networking. Till we resolve this for windows can you try following setup mentioned below.

• Copy following content in docker-compose-raw.yml the arc-dockerized folder
• Relace APPBASE_ID with your APPBASE_ID in appbase container env config
• Run the setup using docker-compose -f docker-compose-raw.yml -up

version: "3.4"
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch-oss:7.9.3
    container_name: elasticsearch
    restart: always
    environment:
      - node.name=elasticsearch
      - discovery.type=single-node
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - http.port=9200
      - http.cors.enabled=true
      - http.cors.allow-origin=*
      - http.cors.allow-headers=X-Requested-With,X-Auth-Token,Content-Type,Content-Length,Authorization
      - http.cors.allow-credentials=true
      - network.publish_host=_local_
      - network.host=_local_,_site_
      - transport.host=127.0.0.1
      - http.host=0.0.0.0
    ports:
      - "9200:9200"
    networks:
      - elasticsearch
  appbase:
    container_name: appbase
    image: appbaseio/arc:7.37.0
    restart: always
    depends_on:
      - elasticsearch
    environment:
      - USERNAME=foo
      - PASSWORD=bar
      - SET_SNIFFING=false
      - APPBASE_ID=YOUR_APPBASE_ID
      - ES_CLUSTER_URL=http://elasticsearch:9200
    volumes:
      - "appbase-data:/appbase-data"
    ports:
      - 8000:8000
    networks:
      - elasticsearch
  fluentbit:
    image: fluent/fluent-bit:1.5
    user: root
    container_name: fluentbit
    command: /fluent-bit/bin/fluent-bit -c /fluent-bit/etc/fluent-bit.conf
    depends_on:
      - "appbase"
    restart: always
    volumes:
      - "./fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf"
      - "./parser.conf:/fluent-bit/etc/parser.conf"
      - "appbase-data:/appbase-data"
volumes:
  appbase-data:
    external: false
networks:
  elasticsearch:
    driver: bridge

[saurabhPV]

how do I obtain appbase id ? I tried and , I also tried username part of it.

[siddharthlatest]

@saurabhPV If you have a cluster within the appbase.io dashboard, that would be appbase.io cloud running the cluster for you or you using it with Bring Your Elasticsearch mode. If you're looking to run this with Docker (i.e. self-hosting appbase.io), this is the path to follow: https://docs.appbase.io/docs/hosting/byoc/#using-docker. You will have to register an appbase.io instance, and you will receive APPBASE_ID in the e-mail. This is a required value for configuring an appbase.io instance with Docker.

[saurabhPV]

Thanks a lot @siddharthlatest @lakhansamani !!! It is working now however I don't see the search logs. I did as follows

1. Used the appbaseid I got in the email and the docker configuration in the above comment. Now all docker services are up and running.
2. Then I shipped my windowbeat logs to the es- I already had this running - I routed it without the arc proxy(i.e on port 9200 directly) - so as expected it didn't generate any request logs.
3. Then I ran search query on my index via arc proxy (i.e on port 8000) and I got the result. so that it would generate request logs
4. But I see in the arc logs level=error msg="[elasticsearch] : \"*permission.Permission\" not found in request context" file=" entry.go:359"
5. And no logs in fluentbit service.

[saurabhPV]

@siddharthlatest , @lakhansamani could you please help me to understand why it throws permission error and how do I fix it ?

[siddharthlatest]

@saurabhPV The permission error isn't an actual error. We have fixed this (will be out in the next release). Are you able to get see the logs?

[saurabhPV]

@siddharthlatest no I don't see the logs I thought because of this error. Other than this I don't see any more logs in any container.

[saurabhPV]

@siddharthlatest I also noticed , it does create es.json which is used in fluent-bit.conf

could it be the reason I don't see logs ?

[siddharthlatest]

@saurabhPV We tried replicating the above issue. While doing that, there was another issue that we saw with how nginx wasn't setting CORS headers. After fixing this, I can see that all the services work well. I'm running the following command: docker-compose -f docker-compose-with-elasticsearch.yaml up -d

The es.json file gets created in the appbase container.

And the logs are getting indexed and visible in the appbase.io dashboard.

Can you try with the updated files here and let me know if it works well for you now?

[saurabhPV]

Thanks a lot @siddharthlatest. I am using the updated docker-compose-with-elasticsearch.yaml and it works!!! Unlike earlier now I can see it has created es.json in appbase-data and I don't see the issue with configure app either I was able to set the .env vars using it!

however I could not test request logs because {"error":{"code":402,"message":"Payment required","status":"Payment Required"}}

Do I need the subscription for running open source feature on my local environment too?

[saurabhPV]

@siddharthlatest Also can you please confirm if it capture user's info? worried because I could not find it in the api documentation example response.

[siddharthlatest]

@saurabhPV It would capture Elasticsearch requests / responses, and store them within your Elasticsearch upstream cluster. It wouldn't capture any user info (i.e. cookies) outside of what you send to the endpoint explicitly. There is no separate collection to a third-party service.

For using the open-source version, you can update the Arc image in the docker compose file to appbaseio/arc-oss instead of appbaseio/arc.

[saurabhPV]

@siddharthlatest thanks a lot! It worked!!! I can see the logs now using the logs endpoint.

I expected it would catch the user details since the basic auth is used for an api call. Nevertheless I would like to add this functionality into arc-oss.
If it's not bothering too much (I know I have already ;-) , sorry about that.) would you be able to tell me if it's possible to pick this info from auth header? I am thinking I can sync users from my app to the appbase user or thorough JWT token parse the user info , intercept the request and ingest it in header so it will be available in the logs.