IRTF-PEARG / draft-safe-internet-measurement

draft-safe-internet-measurement
6 stars 1 forks source link

Tobias' review #23

Open mallory opened 12 months ago

mallory commented 12 months ago

I also believe that there are some more fundamental issues that must be resolved.

Authoritativeness of documents

Marwan noted that the document in its current form may cause harm if it is read by a non-domain expert (say: Medicine Professor) when evaluating a request for ethical clearance of a network measurement study. And yes, this does happen.

Reading the reply by Mallory creates the impression in me that there is some form of assumption that what is written in a document will not be read as authoritative if the authors only (outside the document or somewhere in another part of the document the concerned medicine prof just didn't scroll over) phrase things more in terms of their intentions.

This is not really the case; For the domain where the document will be relevant and useful, those nuances will be lost. It is already hard to have security researchers understand the difference between a draft and and RFC, let alone all the other nuances there after. I would hence caution significant care with how the document can be perceived, especially in context for the point below.

Positionality / Perspective

The draft has a strong perspective from the privacy measurement angle, also highlighted by the statement that some points "[are] based on the

Tor network safe measurement guidelines— so, from the community of measurement and internet research."; The Tor community is a very specific (and small) subset of network measurement work, and I would argue that they are more rooted in the PETS community (also looking at the members of the board).

This community, in terms of type of measurements and also in terms of volume, is very different from the network measurement community (ACM SIGCOMM-Bobble), which again is different from the Network Security bobble (IEEE S&P/NDSS/USENIX SEC/ACM CCS).

These communities have very different requirements and perspectives on network measurements (and potential harm they may cause).

Hence, i'd argue that the draft needs a lot more engagement with these different communities, and should include the experiences of researchers doing that kind of work as well.

Superficial Nature of Sections 2.4ff

From Section 2.4 the draft gets extremely thin and ignores practical requirements of network measurements by taking a strong data minimization perspective (which is imperative for censorship measurements, but often harmful for large scale measurements; If you do a large scale measurement and stripped a bit too much, redoing the measurement might be unnecessary harm; The same as having done an effectively useless measurement.).

Similarly, the--somewhat simon-says-y--headlines thereafter just keep introducing shorter and shorter sections that hand wave away critical aspects that necessitate very careful consideration and instruction... and--as said before--sometimes these imperative things may not even make sense.

2.6 and 2.7 are then close to dangerously short. A document claiming to provide guidelines should provide, for these critical mechanics, actual guidelines on how to, e.g., execute a harm benefit analysis. With the current form, the document essentially provides a blanket for researchers being like "ah, well, thought about it, best we could... uhm... ups, failed after following best practices, nothing we could have done". See also my paper from anrw at 117.

Summary

In summary, as those speaking earlier, I am deeply convinced that the topic of the draft is direly needed; However, such a document should take a lot of context (communities etc.) into account which is not in - 08 (and i would argue what is missing is a multiple of what is already there). It must also provide guidelines on how to do some of the things suggested, instead of just throwing in words, hoping that it is feasible.

Otherwise, this document might cause harm by creating unnecessary red- tape, which will result in losing a lot of the progress that was made in researchers' consideration of ethical concerns and adherence to ethical principles; There will be frustration and 'trying to get around'.

mallory commented 9 months ago

Perhaps also see https://pure.mpg.de/rest/items/item_3517635/component/file_3517636/content