Best search mechanism for Wikipedia offline snapshots

costgallo commented 2 years ago

Prize Bounty

2 winners of 3000 BZZ each

Challenge Description

Swarm decentralized, peer-to-peer storage opens up new opportunities for establishing a self-sovereign cloud. It also brings about a paradigm shift on how to use such storage.

For this bounty we challenge you to create a compelling search mechanism for wikipedia offline snapshots, following this procedure:

Extract the content of any wikipedia ZIM snapshot and use that as dataset (moderately large from https://dumps.wikimedia.org/other/kiwix/zim/wikipedia/ ).
Build a script that based on the snapshot html files content creates some search mechanism under a dedicated html page and/or as injected script in each of the snapshot html files. The search should not use any external online services or OS features. It should work fully offline with resources available in modern web browser.

We will judge the quality of the search mechanism (the more relevant results especially with typos the better), size of any index file (the smaller the better) and user experience of the search.

This bounty is in partnership with Kiwix

Submission Requirements

Open source code license
- Projects must be licensed under an Open Source Licence: i.e., any license for the project shall include terms substantially similar to the terms of version 1.9 of the Open Source Definition, promulgated by the Open Source Initiative at https://opensource.org/osd-annotated.
Documentation of solution: how to run, how it works.
Recorded demonstration of working solution.
A working link to a public Github or Gitlab repository containing the code, team members, presentations, demo and documentation, licence information.
~~This bounty requires a POC or an analysis on how to automate Wikipedia snapshot uploading on Swarm.~~
~~Using Swarm as underlying storage: second layer solutions such as Fairdrive / FairOS and Freezr are applicable.~~

Submission deadline

19th March 2022, 24:00 UTC

Judging Criteria

Meeting the listed requirements.
Compliance with Fair Data Society Principles.
Quality of implementation: code, documentation, technical excellence.
Quality of user experience.
Innovation of implementation and of use case.
Technical complexity of implementation.

Winner Announcement Date

Winner(s) will be announced on 21st March 2022.

tnkrxyz commented 2 years ago

@fairDataSociety I am interested in the task. Can you clarify the last two submission requirements? It seems they are more relevant to your Best upload of Wikipedia mirror to Swarm bounty than this one. Do you also require this bounty task to automate snapshot uploading on Swam & using Swarm as storage? Since the script developed is expected to "work fully offline", it seems at odds with these two requirements. Thanks!

vojtechsimetka commented 2 years ago

Hi @tnkrxyz you are right. The last two requirements are not relevant for this task. I will get them removed. That being said, the search should still work when uploaded to Swarm (=being served as static website)

gitcoinbot commented 2 years ago

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

Work has been started.

These users each claimed they can complete the work by 12 hours from now. Please review their action plans below:

1) r0qs has started work.

Search engine for wikipedia pages hosted on swarm. 2) tnkrxyz has started work.

Search & implement the best offline search solution for Wikipedia snapshot, in response to "Best search mechanism for Wikipedia offline snapshots" hackthon. 3) pro-gramm-er has started work.

I will create a wikipedia search tool. 4) asabya has started work.

I will try to create an indexed search on wikipedia content hosted on swarm 5) pizzooid has started work.

I will develop a search tool for the Wikipedia snapshots, that is usable offline. 6) shepherliu has started work.

a simple offline wiki search dapp on swarm

Learn more on the Gitcoin Issue Details page.

tnkrxyz commented 2 years ago

@vojtechsimetka Makes sense. Thanks for the clarification!

gitcoinbot commented 2 years ago

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

Work for 6000.0 BZZ (7440.0 USD @ $1.24/BZZ) has been submitted by:

@fairdatasociety-github please take a look at the submitted work:

PR by @pizzooid
PR by @r0qs
PR by @shepherliu
PR by @asabya

Learn more on the Gitcoin Issue Details page
Want to chip in? Add your own contribution here.
Questions? Checkout Gitcoin Help or the Gitcoin's Discord
$4,758,480.12 more funded OSS Work available on the Gitcoin Issue Explorer

gitcoinbot commented 2 years ago

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

The funding of 6000.0 BZZ (9480.00 USD @ $1.58/BZZ) attached to this issue has been approved & issued to @asabya.

Learn more on the Gitcoin Issue Details page
Questions? Checkout Gitcoin Help or the Gitcoin's Discord
$5,639,902.89 more funded OSS Work available on the Gitcoin Issue Explorer

fairDataSociety / WAM