Human Score on the Dataset

Simsso / Vision-Based-Page-Rank-Estimation

Student research project on pagerank estimation with deep graph networks

MIT License

5 stars 1 forks source link

Human Score on the Dataset #43

Closed Simsso closed 5 years ago

Simsso commented 5 years ago

We want to determine the human performance on the datasets to compare our model with them.

Simsso commented 5 years ago

Development of the needed tool in the branch human-eval.

Simsso commented 5 years ago

I have made some progress during the bus-ride hackathon last week. Next up is the implementation of a small database (probably just a JSON file) and user identification so results are not lost once a user reloads the page in their web browser.

ghost commented 5 years ago

to do (@Simsso): Add skip button to skip websites, which are already known to the test person. to do (@doktorgibson): Add significance test after evaluation

Sample Size: 100 websites to rank
When: 1st of April
Whom: Valentin, Johannes, Marius, Tobias, Samed, Timo, Felix

Simsso commented 5 years ago

Summary of the needed features as of commit https://github.com/Simsso/Pagerank-Estimation-Deep-Graph-Networks/commit/a1f0ad8e4bec9298db18e2d3a076dcc6b887d872

[x] Endpoints for dataset v2
[x] UI for dataset v2
[x] Server-side storage of results (or client-side with local storage)
[x] Skip button (for known websites)
[x] Instruction text
[x] Enhance user experience with hotkeys

ghost commented 5 years ago

Scores from 1st of April:

A: 60 %

Criteria:

Positive:

Responsive pages
News sites
Personal Sites of seemingly famous people
Modern-looking sites
High content density
Video game sites
Offical-looking sites

Negative:

Placeholders
Error messages
Unsexy design
Unresponsive sites
Personal sites of seemingly unpopular people
Low content density

B: 64%

Criteria: News-Sites ranks higher than blogs, Domain-Parking, Discontinued, 404 rank

C: 58%

Criteria: How much text the website showed, more better, if it looked like a article, overall design matching colors, how professional a website looks

D: 57% (slow); 62% (fast)

Criteria: see B

Simsso commented 5 years ago

@doktorgibson

I suggest moving this to a wiki page because it's something we probably want to keep in the repo and an issue is rather hidden.
Independent from that, I'll extend the issue comment / wiki page with more results from family members this weekend.
I suggest adding the names of the test people because it lets us separate between tech/non-tech people later on, if we ever want to do that. (or age, or gender, whatever)
For each result we should write down how many page pairs the test person has looked at.

ghost commented 5 years ago

Created Wikipage, feel free to edit and add new scores: https://github.com/Simsso/Pagerank-Estimation-Deep-Graph-Networks/wiki/Human-Evaluation-on-PageRank-Estimation

ghost commented 5 years ago

Closing.