Feature: Analytics Endpoint V1

[villanuevawill]

This is the first bounty in an upcoming set of bounties focused around building an analytics dashboard for projects that utilize the standard bounties contract such as gitcoin, and other upcoming open source sites.

This first task begins the work of exposing endpoints around daily stats for a range of time. Ultimately, these will be used to build a react frontend connected to highcharts or other charting libraries. The data is already stored in PSQL, it just needs to be served up via the format requested below. This task ultimately requires just one endpoint to be built for now.

Requirements

• endpoint should accept a start and end-date filter - ?publish_date__range=2016-01-01,2016-02-01 format
• Results need to be returned as daily. Daily represents 00:00 - 23:59 UTC time. The current day will of course be a partial day.
• endpoint should live in the newly added analytics directory/application - https://github.com/Bounties-Network/BountiesAPI/tree/master/bounties_api/analytics
• Url should be prefaced with /analytics route
• A new model should be introduced. This model should include a date field, along with the names of the requested stats. For now, I only foresee one model being created.
• A management job should be created. This job should be intended to run daily. Don't worry about setting up the scheduling - I'll put that work in after the PR. To see an example job/management command: https://github.com/Bounties-Network/BountiesAPI/blob/master/bounties_api/std_bounties/management/commands/get_token_values.py
• The job should fill in empty dates. For example, if the model and table is newly created and empty, the job should insert records for every day. Basically, the job should do calculations from the latest insertion date, but should not repeat work if the date has already been included.
• The endpoint should be an APIView - similar to these - https://github.com/Bounties-Network/BountiesAPI/blob/master/bounties_api/std_bounties/views.py#L83
• The job should already have the previous days pre-calculated. However, the endpoint will need to calculate the statistics for the current day (since it is ongoing) and add that to the dataset (hint: a shared function should be used here)
• This is time-series data! This means, the endpoint should return the values on each stat for each day in-between the range. IN addition to the timeseries data - it should also return the aggregates.
• Pep8 compliance required

Statistics

• Total number of bounties issued
• Total number of fulfillments submitted
• Total number of fulfillments accepted
• Total number of fulfillments without acceptance
• Total fulfiller acceptance rate
• Total bounty fulfilled rate
• Average fulfiller acceptance rate
• Average bounty fulfilled rate
• Average fulfillment amount (for now just eth volume, ignore noneth on this one)
• Total fulfillment amount
• Number of bounties in each bountyStage (each stage count would be a separate field on the model)

All of the above should be a separate field on one model that the job writes to. For the current day, this should be calculated in realtime. These should all be exposed on one api endpoint.

Definition of Done

• All requirements are completed above with the endpoint functional.

Reviewers

Myself and @mbeylin

Review Requirements

• A WIP pull request along the way with initial progress
• Key questions posted here
• Conversational and longer discussions should be directed to the bounties network slack in this channel: https://bountiesnetwork.slack.com/messages/community-dev

[gitcoinbot]

This issue now has a funding of 0.33 ETH (177.45 USD @ $537.71/ETH) attached to it.

• If you would like to work on this issue you can claim it here.
• If you've completed this issue and want to claim the bounty you can do so here
• Questions? Get help on the Gitcoin Slack
• $3866.41 more Funded OSS Work Available at: https://gitcoin.co/explorer

[leonprou]

Hey I like this feature, I started working in a draft, can you check that I understood the solution correctly? Mostly the models.py file. My fork: https://github.com/leonprou/BountiesAPI/tree/analytics_endpoint

[villanuevawill]

@leonprou just looked through it - looks like the right direction. publish_date might be clearer as just date or stats_date, etc... but yeah direction looks right.

[leonprou]

@villanuevawill cool , thanks for quick response 👍 I've thought a little about the job. I have a question about it, when I create a statistics for some day, is this statistics object is immutable? Can some event happen that will require recalculate the statistics for that day? For example if some bounty changes its stage from Draft to Active, the "Number of bounties in each bountyStage" needs to be recalculated.

[villanuevawill]

@leonprou good q. It should be immutable since it is based on the immutable record stored on the blockchain for the contract. Of course, the current day will be mutable, but as described in the task, this can't be pre-calculated for clear reasons.

[villanuevawill]

All records in psql are just mirrors of what is in the blockchain. We're not storing anything unique on our end.

[vs77bb]

Hi @leonprou mind claiming this one on Gitcoin here to show you've started work? https://gitcoin.co/issue/Bounties-Network/BountiesAPI/3

Hope you both are off to a good start this week!

[gitcoinbot]

Work has been started on the 0.33 ETH (156.66 USD @ $474.72/ETH) funding by:

1. @leonprou

   Please work together and coordinate delivery of the issue scope. Gitcoin doesn't know enough about everyones skillsets / free time to say who should work on what, but we trust that the community is smart and well-intentioned enough to work together. As a general rule; if you start work first, youll be at the top of the above list ^^, and should have 'dibs' as long as you follow through.

   On the above list? Please leave a comment to let the funder and the other parties involved what you're working, with respect to this issue and your plans to resolve it. If you don't leave a comment, the funder may expire your submission at their discretion.

   • Learn more on the gitcoin issue page
   • Questions? Get help on the Gitcoin Slack
   • $2607.16 more Funded OSS Work Available at: https://gitcoin.co/explorer

[leonprou]

@villanuevawill Not sure about psql, but looking at the contract - https://github.com/Bounties-Network/StandardBounties/blob/master/contracts/StandardBounties.sol, I see a lot of methods that mutate the bounty. I'll try to catch you on slack tomorrow.

@vs77bb done 👍

[villanuevawill]

@leonprou - I may have misunderstood your question. By immutable, I meant the historical record for the stats will not change. There is a creation date on all of them which is grabbed from the contract. https://github.com/Bounties-Network/BountiesAPI/blob/master/bounties_api/std_bounties/models.py#L49

All the values you need will be in psql - except for acceptance date. However, we'd need to add an accepted_date field to the fulfillment model. The acceptance date can be passed through this func https://github.com/Bounties-Network/BountiesAPI/blob/master/bounties_api/std_bounties/management/commands/bounties_subscriber.py#L87 via event_timestamp which is the timestamp derived from the transaction.

In general there's complexity here, so a convo on slack may clear things up if you have q's. However, you should not need to go to the contract to get the data.

[vs77bb]

@leonprou @villanuevawill Poking in... you guys still working on this? Let me know if you need me to pub this out 🙂

[leonprou]

@vs77bb hey, for now I'm not working on this. Me and @villanuevawill have been discussing this for about a week, and have come to conclusion that some refactoring is needed before proceeding. So now I'm working on this task - https://github.com/Bounties-Network/BountiesAPI/issues/11 to store the events in the db, then I was thinking to work on some sort of event sourcing.. You're invited to join, there's enough work for all I suppose 😄 .

[vs77bb]

@leonprou No worries, that sounds good 👍 was just checking in from Gitcoin! Good luck with #11 in the meantime.

[villanuevawill]

@vs77bb actually, this should not be worked on until some of the work @leonprou does is done. Ideally would like to put this on pause for a bit...

[gitcoinbot]

The funding of 0.33 ETH (131.26 USD @ $397.76/ETH) attached to this issue has been killed by the bounty submitter

• Questions? Get help on the Gitcoin Slack
• $2882.93 more Funded OSS Work Available at: https://gitcoin.co/explorer