User PUT requests an upload to an existing bucket (already happening in threadit but it isn't logged into Celery)
measure size of the put data as it passes through, time of arrival, services sent to, etc. (these metrics aren't measured yet, this will probably go into threadit
load a bucket of user usage into memory, primary user facing key = billing.serverid.username (this will go into billing task functions
update values (billing task function)
write values back whenever values in memory change (billing task function)
aggregate costs when the user asks for the cost (billing task function, this may require results, or possibly using a callback mechanism?)
Notes:
Tracks storage time (time between PUT and DELETE requests)
currently measures data by loading all of it and passing through. Eventually this could be done by streaming to reduce one-time memory usage (say for 4gb file)
Each node maintains its own bucket for each user that it sees. The usage data is monotonically increasing, so the sum of the usage remains the correct value regardless of which node stores the data.
Aggregate by listing out bucket names, then taking buckets with the billing.serverid.username pattern and filtering by username. Then pass aggregated usage through cost functions to estimate cost. This estimated cost is written to a specific bucket (billing.username ?) that contains a bunch of objects, where each one is a cost report for a different aggregation request.
A Django web page view can be made that loads all of the files from the billing.username bucket and presents them as a series of pages that can be viewed to see incurred costs (and pay them?)
Typical example:
threadit
but it isn't logged into Celery)threadit
billing
task functionsbilling
task function)billing task function
)billing task function
, this may require results, or possibly using a callback mechanism?)Notes:
billing.username
?) that contains a bunch of objects, where each one is a cost report for a different aggregation request.billing.username
bucket and presents them as a series of pages that can be viewed to see incurred costs (and pay them?)