Add automated email and URL checker

[palemieux]

Should the system regularly send emails to the registered address and check whether URLs exists and are on black lists?

[dcbullock]

1) An entry that includes a URL shall also include an email address. The email address may be exposed on the website if the controlling entity desires it to be. The URL shall be exposed.

2) All URLs and email addresses shall be updated or reaffirmed periodically by the controlling entity. A reminder shall be sent to the email address in the record. If no update or reaffirmation occurs in a suitable amount of time, any URL and email fields shall be hidden from the website, but shall remain in the database.

3) Reminders to reaffirm shall continue to be sent until an SMTP delivery error is received in return. At such time, reminders shall be disabled for the record, but the record shall remain in the database. (Could skip this step and just reminders forever.)

Perhaps every 6 or 12 months for reaffirmation. This is akin to domain name registration.

If we feel that we need a defense against publishing hostile URLs, we would have to also add a URL blacklist checker.

[SteveLLamb]

For the record I am adamantly against just full on removing Website/email info wholesale without proper validation. This was a very much requested item during the update to the registry, in both ISDCF meetings, and by personal requests.

This was also discussed and agreed on that we'd not validate or remove URLs/emails. And all updates were the responsibility of the owner. (https://github.com/ISDCF/registries/issues/53).

[dcbullock]

Our comments might have the same basic requirement:

1. those that register information are responsible for keeping it up to date.

I think this issue is adding a new requirement:

1. registered information should not be allowed to a. refer to blacklisted URLs b. go stale - refer to non-responsive email addresses or URL

The 2b requirement is possibly not the goal - I might be making that up.

If we are not trying to build for 2b, then a simple periodic check of URLs against a blacklist will meet the requirement.

[SteveLLamb]

Our comments might have the same basic requirement:

Right. My comment was more to the knee jerk reaction during the meeting to just removing the info from the site completely as an option.

[jamiegau]

There was a mention of pay per year for being listed. Just like domain names. I think this would be a good idea. Keeping it cheap, like $20usd per year. as it would stop a lot of registrations that are likely "Just because I can". Or because I can put a blacklisted URL in there and it costs nothing. Also, as ICANN domains are not free - as it takes considerable work to keep that all up to date. So why not have the Intersociety derive a small income from it like ICANN from domain names?

At $20usd, it is so cheap, I would just pay it and move on.

Also, adding something like a PayPal link to a website is not that hard from the last time I looked.

In my opinion, ICANN and domains are a central body that keeps core infrastructure organised and independent. I feel ISDCF should look at doing a similar job for Digital Cinema.

[dcbullock]

I think the question of paying is not on topic for this issue. I am sure that ISDCF will continue to discuss that going forward, but this is not the right forum for it. @jamiegau can you put your comment above on the reflector for full ISDCF discussion?

[dcbullock]

Our comments might have the same basic requirement:

Right. My comment was more to the knee jerk reaction during the meeting to just removing the info from the site completely as an option.

Understood. It sure sounds like this issue could be resolved with what I assume is straightforward periodic blacklist check as @palemieux noted in the description.

Other things could be attached to future Feature requests.

[japierce1028]

How hard is it to test the URL against a blacklist registry automatically?

The number of facility and studio entries that have an email address is under 200 so if we charged $20usd it would be maximum of $4Kusd - assuming we got 25%, it hardly seems worth it. I think it's more valuable to have entries than not for users of the system, so I don't think a barrier to have a name listed would be smart.

So I'm leaning to keeping the list showing both URL and email. Automatically check the URL against a blacklist and remove those that bounce. Keep the email - because it hardly hurts if someone sends an email and it bounces...

[jamiegau]

There are free tools to check for different types of blacklisting. Check this https://stackoverflow.com/questions/17777085/api-for-checking-blacklisted-ip-addresses for a general overview.

The problem would be more on which one to use.

[palemieux]

I would think we would want to start with: https://developers.google.com/safe-browsing/v4

Couple of questions:

• if a URL fails the test, does it:

  • get removed from the registry altogether
  • get flagged in the registry
  • not get shown on the web page

• how often does this check happen?

[japierce1028]

I would think the URL should be flagged and not shown on the web page. I could easily be convinced that it should be removed altogether, since we have a way to track what it was from both the GitHub history and the google form input.