search

AISG-Technology-Team / AISG-Online-Safety-Challenge-Submission-Guide

Submission Guide + Discussion Board for AI Singapore Online Safety Prize Challenge
https://ospc.aisingapore.org
14 stars 5 forks source link

offensive memes - aliencaocao - request for logs #45

Closed aliencaocao closed 8 months ago

aliencaocao commented 8 months ago

Hi, we had a time limit exceeded error and we would like the logs to know why. fed3a4b8-def2-4dba-bdba-5e4eeab4f05b

AISG-Wesley commented 8 months ago

We will not be releasing the logs for this submission. The logs contain information that exposes elements of the dataset.

aliencaocao commented 8 months ago

Can we kindly request clarification on which exact part is exposing elements of the dataset? We cannot focus on the challenge task itself when we do not get any feedback from a black-box runtime and without any information on what kind of dataset information are we even exposing, it is impossible to use stderr as the only source of feedback. I understand if you feel the logs are sensitive, but I don't understand why it is also sensitive to just let us know which part of our log is offending so we can remove them.

Really hope to participate seriously but with the existing rules, it is becoming very hard.

AISG-Wesley commented 8 months ago

We understand your frustration, but we seek your understanding of our position in ensuring a fair and reasonable challenge for all.

Without necessarily being exhaustive, we expect logging information to stderr to not show any information that was provided in stdin, and to not show any information that ought to be shown in stdout.

Do note that while the dataset that the Docker containers are executed on are hidden, the contents of the Docker containers themselves are wholly under the control of the participant. We thus make the stand that participants are wholly responsible for what their containers are doing.

aliencaocao commented 8 months ago

Ok, we will minimize the logs further and try again. Thanks for the explanation

aliencaocao commented 8 months ago

Just to be clear, is printing an 'Error processing response for image id {id}' considered sensitive?

AISG-Wesley commented 8 months ago

If the supposed id can be interpreted as any form of matching with any subset of the dataset, then it is sensitive.

aliencaocao commented 8 months ago

So like a numerical id from 0 to 1700 is counted yes, while a uuid isnt?

mtmak-aisg commented 8 months ago

So like a numerical id from 0 to 1700 is counted yes, while a uuid isnt?

Devoid of context, this cannot be answered. Please make a submission with your proposed updates to stderr and request for the logs. At that point, the outcome should be clearer.