Improve readme documentation on how to provide a new crawler

hugolpz commented 3 years ago

This /CONTRIBUTING.md is a License Agreement / Code of Conduite to sign. As far as I can see, this very cherishable project has no actual tutorial.

In don't have Python and code knowledge to fix this documentation issue but I can map the road so it become easier for the next person to do so.

Wanted

If an user want to add a language such as Catalan from Barcelona (ca, cat : missing). What do he needs to jump in quickly ? What should he provide ?

What isn the local structure :
- util.py : store functions uses by multiple languages crawlers
- main.py : stores the 1000+ crawlers calls, run them all.
- crawl_{iso}.py : stores language-specific copora's source url and processing functions.
  - crawl_ca_valencia.py
What tools I have :
- lists of available modules
- API of key functions
What input(s) : python list of url ?
What are the classic parts of a crawler function ?
What output format : raw text ? html is fine because a html balise wiper is then used ?
Example of easily hackable base-code.

API (to complete)

Defined functions within util.py, by order of apparition as of 2021/02/26. If you have some relevant knowledge, please help for a sub-section or one item.

Some tools

daterange(start, end): __
urlpath(url): __
urlencode(url): __

Main element

class Crawler(object):
- __init__(self, language, output_dir, cache_dir, crawldelay): __
- get_output(self, language=None): __
- close(self): __
- fetch(self, url, redirections=None, fetch_encoding='utf-8'): __
- fetch_content(self, url, allow_404=False): __
- fetch_sitemap(self, url, processed=set(), subsitemap_filter=lambda x: True): __
- is_fetch_allowed_by_robots_txt(self, url): __
- crawl_pngscriptures_org(self, out, language): __
- _find_urls_on_pngscriptures_org(self, language): __
- crawl_abc_net_au(self, out, program_id): __
- crawl_churchio(self, out, bible_id): __
- crawl_aps_dz(self, out, prefix): __
- crawl_sverigesradio(self, out, program_id): __
- crawl_voice_of_america(self, out, host, ignore_ascii=False): __
- set_context(self, context): __

Some crawlers for multi-languages sites

crawl_bbc_news(crawler, out, urlprefix): __
crawl_korero_html(crawler, out, project, genre, filepath): __
write_paragraphs(et, out): __
crawl_deutsche_welle(crawler, out, prefix, need_percent_in_url=False): __
crawl_radio_free_asia(crawler, out, edition, start_year=1998): __
crawl_sputnik_news(crawler, out, host): __
crawl_udhr(crawler, out, filename): __
crawl_voice_of_nigeria(crawler, out, urlprefix): __
crawl_bibleis(crawler, out, bible): __
crawl_tipitaka(crawler, out, script): __
find_wordpress_urls(crawler, site, **kwargs): __

Some cleaners

unichar(i): __
replace_html_entities(html): __
cleantext(html): __
clean_paragraphs(html): __
extract(before, after, html): __
fixquotes(s): __

Shorter way to do so

In code comments can do a lot. Pointing to wisely chosen sections too. If you have the required know how, please add comments onto a chosen, existing crawler and point to it as an in-code tutorial.

@sffc, @brawer : anyone could help on that ?

Aayush-hub commented 3 years ago

@hugolpz Can I help adding steps to readme about how to add new crawler starting with basics of installing python?

hugolpz commented 3 years ago

Hello Aayush. thank you for jumping in. I think we can assume ability to install python. Readme.md should just have a section "Requirement" with python version and associated pip dependency

### Requirements
* python x.x+

### Dependencies
`
pip3 instal {package1}
pip3 instal {package2}
pip3 instal {package3}

This would help yes.

I made a large review of this project but I'am JS dev so I walk quite blind here. Yet I think this project isn't that hard to contribute to : the main obstacle is 1. how to start and 2. what kind of output each crawler must provides, how, where.

@brawer, would you temporarily grant me maintainer status so I could handle the possible PRs ? I would be happy to give that userright back as soon as a new, active python dev emerges.

Aayush-hub commented 3 years ago

@hugolpz Sure, looking to add required dependencies information in README :)

Aayush-hub commented 3 years ago

@hugolpz Getting an error no module found : corpuscrawler when running main.py. Can you please help debugging it?

hugolpz commented 3 years ago

JS dev here, I try to help around but I don't know python. I can look for python help but it will need at least 5 days.

google / corpuscrawler