Upload un-submitted reports with a tighter interval

[hellais]

Currently un-submitted reports are only being uploaded every 24 hours. I would suggest we make that happen more often, since it doesn't add any extra cost to check if there are un-uploaded reports and it allows us to trigger uploads with a tighter interval.

This would avoid situations where the network is not working consistently at the daily schedule.

I would say we can trigger this cronjob every 15 minutes.

[anadahz]

Yes it makes sense to increase the trigger interval of reports uploading, we should though evaluate the following cases prior of introducing a short upload interval:

1. Concurrent upload of reports. On slow internet connections there may be a number of triggered upload report tasks that could potential re-upload the same un-submitted reports. This can be easily fixed with a file lock but still the cronjobs will queue up and waiting to be run.
2. Bandwidth consumption of invalid reports upload. There are certain cases where reports can't be completely uploaded and are not tagged as incomplete triggering the upload process of oonireport. This occurrence will significantly increase the bandwidth consumption of the probe.

Does it makes more sense to have this interval being triggered every 12 or 7 hours instead?

[hellais]

Regarding 1. you can just make the lock fail when it is already acquired, the flock option for this is --nonblock.

Regarding 2. if this is a problem of oonireport it should be fixed in there rather than having workarounds for it in the raspberry pi image.

I think an interval of 12 hours or 7 hours is definitely too much as what we are seeing in the field is that often times people don't have the pi plugged in for that long and we should manage to get the measurements from it.

[anadahz]

@hellais:

1. I wouldn't recommend using --nonblock since it can result to reports not being run at all. There have been a number of ooniprobe measurements that have been stale, for instance due to network connectivity issues.
2. This needs further investigation but it's very hard to debug and catch these issues.

If we want to ensure that cronjobs are still running in non 24 hours operation we can instead use anacron that assumes a system is not running continuously and can execute cronjobs periodically in days instead of hours.

Note that Lepidopter is being designed for 24 hour operation, non maintained shutdowns such as power failures or disconnection from the mains will wear out SD cards way faster than normal and bring Lepidopter in an unusable state.

[hellais]

1. I wouldn't recommend using --nonblock since it can result to reports not being run at all.

Why would this lead to reports not being run at all? To be clear I mean the lock should only be on the oonireport command and not on ooniprobe.

There have been a number of ooniprobe measurements that have been stale, for instance due to network connectivity issues.

What do you mean by "measurements that have been stale"?

I understand the concerns around "non maintained shutdowns", but given the reality of how it is being used we should try to take into account these.

[anadahz]

Why would this lead to reports not being run at all? To be clear I mean the lock should only be on the oonireport command and not on ooniprobe.

You are right I was confused with the ooniprobe, yes this this can be included in oonireport but given the fact that we want to protect against improper shutdowns anacron may be a better fit.

What do you mean by "measurements that have been stale"?

Hanged ooniprobe/oonireport processes. Not sure under which circumstances this is triggered but I remember seeing on a Lepidopter platform on the field where I had no time at that point to investigate.

I understand the concerns around "non maintained shutdowns", but given the reality of how it is being used we should try to take into account these.

We should perhaps then prioritize https://github.com/TheTorProject/lepidopter/issues/50. Having an unusable Lepidopter installation due to SD card issues is not useful to anyone.

[anadahz]

This is currently supported in ooniprobe version 2.0.0 and no need to be implemented in lepidopter.