Script - Time series (Geocoder - PUB)

[BK01]

Skills / SW:

• Scripting approach TBD, csv
• DataBC API Gateway (Kong/Cassandra) *Methodology will need to be determined in collaboration with DataBC middleware team

Description: Currently we use Elk to record stats for several APIs (contact LL or BK for URL).

Based on the average 50GB of storage used by logs per month, Elk is used for real-time observations only as well as analytics up to 30 days. Consequently we have a service gap with respect to time series.

However, clients of Elk have thus far only had a requirement to record a time series of monthly totals, rather than individual days. To achieve this clients manually visit the site once a month and record the numbers. If the end of the month falls on a weekend, holiday or staff are away, a number collected late may be incorrect.

Can we write a script to auto query Elk or simply use the underlying logs at the end of each month and append results to a file?

APIs: BC Physical address geocoder

Clients are currently requesting this [WBC, DBC].

Deliverables:

• Script(s) used to query monthly stats for the BC Physical Address Geocoder
• stats.csv -- data to include: month name, total hits

Deliverables - extended:

• Time permitting: Investigate options to generate same stats for other/all APIs using the DataBC API Gateway

[BK01]

Use REST API for Elk. DataBC needs to provide access.

DataBC - to create a job in Jenkins and provide branders config access to update jenkins job by adding the python script, execute, test, refine etc

branders - to write a Python script within Jenkins.

Eventually store the python script in GOGS

[BK01]

Jenkins job created, access granted to BK01 and branders

[BK01]

DataBC to try running against PROD stats. Script currently using TEST. Env passed to script as arg from Jenkins job.