Bounty: Create A CDN Log Analytics Tool

[ericxtang]

Goal: Create a custom CDN log analytics tool for Livepeer video streams.

Scope We want to be able to analyze CDN data usage per video stream.

This tool should contain the following features:

1. Pulls a list of logs down from GCS gsutil -o "GSUtil:parallel_process_count=1" -m cp -r gs://{bucket-name} {local-dir}
2. Analyzes the logs in a local directory. It should iterate through the directory recursively to make sure all log files in sub directories are analyzed. It should be performant, taking advantage of multiple processors on a machine so it can be ran on a high-cpu cloud instance. This tool is meant to be used once-a-month, and the expected dataset can be up to 20GB.
3. Return a report of the total data usage for each stream, as well as a sql query that allows a user to insert the report into a postgres database.

Example commands:

• log-analyzer download {gs-bucket-url}
• log-analyzer analyze {local-dir} -o report.csv
• log-analyzer analyze {local-dir} -o insert.sql
• log-analyzer insert {postgres-db-info}

The report should contain:

• manifest_id (This is in the format of 4950yo5vcetc1cvu)
• stream_id (This is in the format of a2c50bb5-d6d5-481d-a50f-90dece3fae71)
• stream_name (This is in the format of video+4950yo5vcetc1cvu)
• date
• number_viewers (inferred by number of unique client ips)
• total_filesize
• total_cs_bytes
• total_sc_bytes
• url
• response_code

Looking at the example log files, you will see that Livepeer stream URLs are formatted like:

• /hls/{manifest_id}/{rendition}/{segment}.ts (video segment)
• /hls/{manifest_id}/{rendition}/index.m3u8 (rendition playlist)
• /hls/{manifest_id}/index.m3u8 (master playlist)
• /recordings/{stream_id}/index.m3u8
• /hls/{stream_name}/index.m3u8

Each of the log lines also contain the filesize, the client-to-server bytes transferred (cs-bytes), server-to-client bytes transferred (sc-bytes), and the client ip (c-ip).

This tool should not crash. If unexpected URLs appear in the logs - and instead, log a warning message.

Requirements:

• Use golang.
• Contains unit tests.
• Contains documentation on how to use the tool.
• Contains high level performance benchmark (amount of time it took to process different sizes of data, up to 20GB).

[gitcoinbot]

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

---

This issue now has a funding of 0.25 ETH (548.77 USD @ $2195.1/ETH) attached to it.

• If you would like to work on this issue you can 'start work' on the Gitcoin Issue Details page.
  • Want to chip in? Add your own contribution here.
  • Questions? Checkout Gitcoin Help or the Gitcoin's Discord
  • $2,272,093.77 more funded OSS Work available on the Gitcoin Issue Explorer

[gitcoinbot]

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

---

Work has been started.

These users each claimed they can complete the work by 265 years, 7 months from now. Please review their action plans below:

1) baldator has been approved to start work.

I'd like to take this one. I'll implement the cdn log analytics in golang following given specs. 2) agbilotia1998 has applied to start work _(Funders only: approve worker | reject worker)_.

1. Analyze the requirements.
2. Provide customization options like number of threads as CLI arguments.
3. Follow test driven approach for building the tool. 3) samhar1995 has applied to start work _(Funders only: approve worker | reject worker)_.

I would like to apply for this role

I've experience in working with golang and creating analytics tools

Learn more on the Gitcoin Issue Details page.

[gitcoinbot]

@baldator Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

• [x] reminder (3 days)
• [ ] escalation to mods (6 days)

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

[gitcoinbot]

@baldator Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

• [x] reminder (3 days)
• [ ] escalation to mods (6 days)

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

[gitcoinbot]

@baldator Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

• [x] reminder (3 days)
• [ ] escalation to mods (6 days)

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

[gitcoinbot]

@baldator Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

• [x] reminder (3 days)
• [ ] escalation to mods (6 days)

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

[gitcoinbot]

@baldator Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

• [x] reminder (3 days)
• [ ] escalation to mods (6 days)

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

[gitcoinbot]

@baldator Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

• [x] reminder (3 days)
• [ ] escalation to mods (6 days)

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

[gitcoinbot]

@baldator Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

• [x] reminder (3 days)
• [ ] escalation to mods (6 days)

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

[gitcoinbot]

@baldator Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

• [x] reminder (3 days)
• [ ] escalation to mods (6 days)

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

[gitcoinbot]

@baldator Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

• [x] reminder (3 days)
• [ ] escalation to mods (6 days)

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

[gitcoinbot]

@baldator Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

• [x] reminder (3 days)
• [ ] escalation to mods (6 days)

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

[gitcoinbot]

@baldator Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

• [x] reminder (3 days)
• [ ] escalation to mods (6 days)

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

[gitcoinbot]

@baldator Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

• [x] reminder (3 days)
• [ ] escalation to mods (6 days)

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

[gitcoinbot]

@baldator Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

• [x] reminder (3 days)
• [ ] escalation to mods (6 days)

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

[gitcoinbot]

@baldator Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

• [x] reminder (3 days)
• [ ] escalation to mods (6 days)

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

[gitcoinbot]

@baldator Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

• [x] reminder (3 days)
• [ ] escalation to mods (6 days)

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

[gitcoinbot]

@baldator Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

• [x] reminder (3 days)
• [ ] escalation to mods (6 days)

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

[gitcoinbot]

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

---

Work for 0.25 ETH (888.63 USD @ $3554.54/ETH) has been submitted by:

---

• Learn more on the Gitcoin Issue Details page
• Want to chip in? Add your own contribution here.
• Questions? Checkout Gitcoin Help or the Gitcoin's Discord
• $2,037,659.99 more funded OSS Work available on the Gitcoin Issue Explorer