JessyBarrette commented 4 months ago

166 is getting close to be able to running accordingly and need a few fine tuning and clean. One issue left is to be able to track the data downloads via plausible.

We can easily deploy scripts via the init.d folder within an erddap container and I think this would be the ideal method to implement the plausible download tracker.

@fostermh How is setup right now your log scrapper for tracking downloads. Can we include the dependacies within a Dockerfile and add the script via an executable file?

See example of the Dockerfile: here

and list of executable to run on the container start here: https://github.com/HakaiInstitute/hakai-datasets/tree/caprover-deploy/init.d

Any files within init.d set as executable or matching. *.sh will get executed when a container is started.

JessyBarrette commented 4 months ago

After talking with @fostermh

Since the present method @fostermh use is using the appache log on top of the container. We will attempt to log usage by having a second container running in parallel that access the nginx log of the caprover erddap app.

The initial method was using appache, Matt will review if this method is compatible with NGINX which is used by caprover.

JessyBarrette commented 4 months ago

In consequence, we will ignore this present issue while developing #166

fostermh commented 4 months ago

expanding on the details Jessy has already posted.

We will attempt to use telegraf running in a docker container to parse the nginx logs from caprover. This mirrors the current production setup in which erddap is running behind a proxy (apache) and telegraf is used to scrape the apache logs and report back on erddap usage.

TODO:

[x] export nginx logs from caprover nginx container so other containers can access them. See the following for details:
- https://github.com/caprover/caprover/issues/880
- https://caprover.com/docs/nginx-customization.html In short, update nginx config, via caprover setup page, to send access logs to/nginx-shared/
[x] mount nginx logs in telegraf container (/captain/data/nginx-shared/) and adjust log scraper settings as needed
[x] setup plausible page and add key to telegraf container config
[x] setup sentry project and add key to telegraf container config (for cron job like monitoring)
[ ] adjust sentry cron timeout so we don't get spammed and hate matt
[ ] profit.

https://github.com/cioos-siooc/cwatch-telegraf

steviewanders commented 4 months ago

👌🏼

-- Steve Vandervalk Hakai Institute

On Thu, 15 Feb 2024 at 12:44, Matthew Foster @.***> wrote:

expanding on the details Jessy has already posted.

We will attempt to use telegraph running in a docker container to parse the nginx logs from caprover. This mirrors the current production setup in which erddap is running behind a proxy (apache) and telegraph is used to scrape the apache logs and report back on erddap usage.

TODO:

export nginx logs from caprover nginx container so other containers can access them. See the following for details:

caprover/caprover#880 https://github.com/caprover/caprover/issues/880

https://caprover.com/docs/nginx-customization.html In short, update nginx config, via caprover setup page, to send access logs to /nginx-shared/

mount nginx logs in telegraph container (/captain/data/nginx-shared/) and adjust log scraper settings as needed

setup plausible page and add key to telegraph container config

setup sentry project and add key to telegraph container config (for cron job like monitoring)

adjust sentry cron timeout so we don't get spammed and hate matt

profit.

https://github.com/cioos-siooc/cwatch-telegraph

— Reply to this email directly, view it on GitHub https://github.com/HakaiInstitute/hakai-datasets/issues/172#issuecomment-1947312404, or unsubscribe https://github.com/notifications/unsubscribe-auth/ARESN62WAR6CKJADNCB6LITYTZXULAVCNFSM6AAAAABDKSKI6SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNBXGMYTENBQGQ . You are receiving this because you were assigned.Message ID: @.***>

JessyBarrette commented 4 months ago

Just to add to present thread. Looks like ERDDAP itself is also suggesting to use the Tomcat/Apache/(NGINX?) log to track statistics and usage. https://coastwatch.pfeg.noaa.gov/erddap/download/setup.html#tomcatLogs

@fostermh not sure if you started developing this after reading the ERDDAP docs but this is reassuring to see :)

fostermh commented 4 months ago

I had not read the docs but yes nice to see that we independently arrived at the same solution. Hopefully we are on the correct track. :-)

steviewanders commented 3 months ago

@fostermh Thanks, I'll give this a shot on the development instance next week.

steviewanders commented 2 months ago

Other options I've used to parse NGINX - out of the box - in the past. These need to be ruled out before a custom setup is attempted.

steviewanders commented 2 months ago

[x] https://goaccess.io/

https://github.com/HakaiInstitute/erddap-goaccess

There is an example report.html in there we will need to check against requirements.

fostermh commented 2 months ago

telegraf is scraping logs for the development version of erddap

telegraf container has been setup https://captain.erddap.hakai.app/#/apps/details/telegraf
pushing to plausible https://plausible.server.hakai.app/development.erddap.hakai.app
and is checking in with sentry https://hakai-institute.sentry.io/crons/hakai-telegraf-erddap/hakai-erddap-telegraf-checkin/

steviewanders commented 2 months ago

Amazing "scrapping".

At the next ERDDAP 2.0 meeting lets compare and contrast and figure out what next or if this is oh so done.

fostermh commented 2 months ago

haha spelling corrected. Yes would be good to contrast, sounds good.

JessyBarrette commented 2 months ago

I just had a quick look at the plausible and for some reasons the urls it refers to have a section doubled:

https://development.erddap.hakai.appdevelopment.erddap.hakai.app/erddap/tabledap/HakaiBamfieldBoL5min.htmlTable

Must be something related to the setup with plausible.

fostermh commented 2 months ago

sorted. the host_url environment variable must contain the protocol, which I had forgotten to include in the telegraf setup. Thanks for noticing.

JessyBarrette commented 2 months ago

I just ran sucessfully on development.erddap.hakai.app/erddap the cde harvester. You should now have logged a number of different csv querries to the erddap. :)

I think we should have all we need now

steviewanders commented 2 months ago

Nice! Lets meet at the end of this week and review these two analytics quick.

steviewanders commented 2 months ago

[ ] @steviewanders Needs to add this to an analytics section of the resulting documentation & diagram but basically:
[x] https://github.com/allinurl/goaccess works to see all requests NGINX serves, which allows us to sort between monitoring bots, our own API requests (CDE et al.), and normal users of the HTML pages
[x] The specific setup for the above is here https://github.com/HakaiInstitute/erddap-goaccess
[x] It relies upon a NGINX access.log being persisted outside the ERDDAP container to the EC2 file system, which could benefit from a simple backup in case of $something_bad
[ ] Need to add a weekly cron job to generate a report and dump it and the log to S3
[x] Plausible can and has been added to the HTML template for ERDDAP so it runs as a client side Javascript tracker just GA
[x] Combined, these two address our two usage questions around ERDDAP.

HakaiInstitute / hakai-datasets

Tracking downloads with plausible and ERDDAP #172

166 is getting close to be able to running accordingly and need a few fine tuning and clean. One issue left is to be able to track the data downloads via plausible.