I also believe that
there are some more fundamental issues that must be resolved.
Authoritativeness of documents
Marwan noted that the document in its current form may cause harm if it
is read by a non-domain expert (say: Medicine Professor) when
evaluating a request for ethical clearance of a network measurement
study. And yes, this does happen.
Reading the reply by Mallory creates the impression in me that there is
some form of assumption that what is written in a document will not be
read as authoritative if the authors only (outside the document or
somewhere in another part of the document the concerned medicine prof
just didn't scroll over) phrase things more in terms of their
intentions.
This is not really the case; For the domain where the document will be
relevant and useful, those nuances will be lost. It is already hard to
have security researchers understand the difference between a draft and
and RFC, let alone all the other nuances there after. I would hence
caution significant care with how the document can be perceived,
especially in context for the point below.
Positionality / Perspective
The draft has a strong perspective from the privacy measurement angle,
also highlighted by the statement that some points "[are] based on the
Tor network safe measurement guidelines— so, from the community of
measurement and internet research."; The Tor community is a very
specific (and small) subset of network measurement work, and I would
argue that they are more rooted in the PETS community (also looking at
the members of the board).
This community, in terms of type of measurements and also in terms of
volume, is very different from the network measurement community (ACM
SIGCOMM-Bobble), which again is different from the Network Security
bobble (IEEE S&P/NDSS/USENIX SEC/ACM CCS).
These communities have very different requirements and perspectives on
network measurements (and potential harm they may cause).
Hence, i'd argue that the draft needs a lot more engagement with these
different communities, and should include the experiences of
researchers doing that kind of work as well.
Superficial Nature of Sections 2.4ff
From Section 2.4 the draft gets extremely thin and ignores practical
requirements of network measurements by taking a strong data
minimization perspective (which is imperative for censorship
measurements, but often harmful for large scale measurements; If you
do a large scale measurement and stripped a bit too much, redoing the
measurement might be unnecessary harm; The same as having done an
effectively useless measurement.).
Similarly, the--somewhat simon-says-y--headlines thereafter just keep
introducing shorter and shorter sections that hand wave away critical
aspects that necessitate very careful consideration and
instruction... and--as said before--sometimes these imperative things
may not even make sense.
2.6 and 2.7 are then close to dangerously short. A document claiming to
provide guidelines should provide, for these critical mechanics, actual
guidelines on how to, e.g., execute a harm benefit analysis. With the
current form, the document essentially provides a blanket for
researchers being like "ah, well, thought about it, best we could...
uhm... ups, failed after following best practices, nothing we could
have done". See also my paper from anrw at 117.
Summary
In summary, as those speaking earlier, I am deeply convinced that the
topic of the draft is direly needed; However, such a document should
take a lot of context (communities etc.) into account which is not in -
08 (and i would argue what is missing is a multiple of what is already
there). It must also provide guidelines on how to do some of the
things suggested, instead of just throwing in words, hoping that it is
feasible.
Otherwise, this document might cause harm by creating unnecessary red-
tape, which will result in losing a lot of the progress that was made
in researchers' consideration of ethical concerns and adherence to
ethical principles; There will be frustration and 'trying to get
around'.
mallory
commented
10 months ago
I also believe that there are some more fundamental issues that must be resolved.
Authoritativeness of documents
Marwan noted that the document in its current form may cause harm if it is read by a non-domain expert (say: Medicine Professor) when evaluating a request for ethical clearance of a network measurement study. And yes, this does happen.
Reading the reply by Mallory creates the impression in me that there is some form of assumption that what is written in a document will not be read as authoritative if the authors only (outside the document or somewhere in another part of the document the concerned medicine prof just didn't scroll over) phrase things more in terms of their intentions.
This is not really the case; For the domain where the document will be relevant and useful, those nuances will be lost. It is already hard to have security researchers understand the difference between a draft and and RFC, let alone all the other nuances there after. I would hence caution significant care with how the document can be perceived, especially in context for the point below.
Positionality / Perspective
The draft has a strong perspective from the privacy measurement angle, also highlighted by the statement that some points "[are] based on the
Tor network safe measurement guidelines— so, from the community of measurement and internet research."; The Tor community is a very specific (and small) subset of network measurement work, and I would argue that they are more rooted in the PETS community (also looking at the members of the board).
This community, in terms of type of measurements and also in terms of volume, is very different from the network measurement community (ACM SIGCOMM-Bobble), which again is different from the Network Security bobble (IEEE S&P/NDSS/USENIX SEC/ACM CCS).
These communities have very different requirements and perspectives on network measurements (and potential harm they may cause).
Hence, i'd argue that the draft needs a lot more engagement with these different communities, and should include the experiences of researchers doing that kind of work as well.
Superficial Nature of Sections 2.4ff
Similarly, the--somewhat simon-says-y--headlines thereafter just keep introducing shorter and shorter sections that hand wave away critical aspects that necessitate very careful consideration and instruction... and--as said before--sometimes these imperative things may not even make sense.
2.6 and 2.7 are then close to dangerously short. A document claiming to provide guidelines should provide, for these critical mechanics, actual guidelines on how to, e.g., execute a harm benefit analysis. With the current form, the document essentially provides a blanket for researchers being like "ah, well, thought about it, best we could... uhm... ups, failed after following best practices, nothing we could have done". See also my paper from anrw at 117.
Summary
In summary, as those speaking earlier, I am deeply convinced that the topic of the draft is direly needed; However, such a document should take a lot of context (communities etc.) into account which is not in - 08 (and i would argue what is missing is a multiple of what is already there). It must also provide guidelines on how to do some of the things suggested, instead of just throwing in words, hoping that it is feasible.
Otherwise, this document might cause harm by creating unnecessary red- tape, which will result in losing a lot of the progress that was made in researchers' consideration of ethical concerns and adherence to ethical principles; There will be frustration and 'trying to get around'.