Add website URL validation

[schlos]

Where to implement the change: "Morph script" - script that parses TJV register and persists it to db https://morph.io/SelectSoft/blue_gene

Current: Sometimes due to human error, processing is not done properly, skipped or stopped, so resulting Morph.io database has invalid records.

Expected: We want to make sure that Morph.io file has valid items. Add check in Morph.io scraper in Website field:

• website

See also: https://stackoverflow.com/questions/7160737/python-how-to-validate-a-url-in-python-malformed-or-not and https://stackoverflow.com/questions/22238090/validating-urls-in-python

Add website URL validation

After scraping public TJV register, add a RegEx check in website field when value is not null (!=null).

Based on result of a RegEx:

• IF website field passed the URL validation (URL structure is valid), persist URL to the website field
• IF website field failed the URL validation (URL structure is invalid), do not persist URL to the website field (leave website field blank)

[schlos]

@SelectSoft some failed web addresses are failed just because they show long URL (i.e. path to a page), i.e.

http://www.sunja.hr/ustanove/dječji-vrtic-bambi.html https://zlatar.hr/galerija-izvorne-umjetnosti-zlatar/

Could you check if RegEx can account for this case also to mark it as success?

---

Some values are failed but they are correct websites, like:

https://inspektorat.gov.hr/ http://web.reakvarner.hr/

Could you check? is it because of ending slash character?

---

I have one more request, same as in codeforcroatia/imamopravoznati-tjv#5 :

In the fields

email_validation_pass | website_validation_pass | foi_officer_email_validation_pass

currently we have following values:

• true
• fail
• nan

Could we change wording to use same system? Expected would be something like:

• true
• false
• null

This new wording has more sense, right?

Thanks!