Add URL validation to make sure people are linking their /uses page

[emgoto]

Taking a look at some of the open PRs, people have linked their website but not their /uses page. Maybe we could add some sort of eslint rule to make sure the URL is correct?

[transiient]

That won't be possible with just /uses because some people use URLs like abc.com/what-i-am-using or abc.com/what-i-use. Maybe check against a list like use, uses, stack, gear, kit, and using (possibly adding others too depending on what people are using)?

[iamandrewluca]

I think we should add a validator that detects 404 pages also, people are just adding their portofolio, for the sake of being on https://uses.tech/

[wesbos]

agreed - I'd love to just remove anyone who doesn't use /uses, bu that seems not nice haha.

A 404 checker would be nice - anyone want to take a stab at a github action for this?

[HugoDF]

@wesbos happy to have a go

Would it make sense to add it as part of the "validate-data" action or a separate one (I think the use/gear/uses matching makes sense there, not sure about 404s)

[HugoDF]

I'm going to allow URLs that start with https://gist.github.com/ so that these 2 keep working (they're gists)

"url" with value "https://gist.github.com/diurivj/78ca931c4b20dca1e1e13982fa9c309d" fails to match the required pattern: /(use|uses|using|setup|environment)/
"url" with value "https://gist.github.com/zilahir/4aaf5907999ea53711b2d554d22b0f3f" fails to match the required pattern: /(use|uses|using|setup|environment)/

[iamandrewluca]

@HugoDF here is an attemp I started. I think we should not force url ending in /uses. Bellow is adapted to check if url will return status code 200, but it will run for long time if more and more users are added. There need to be a url extract from PR diff and to validate just added URLs

import Joi from '@hapi/joi';
import core from '@actions/core';
import { promisify } from 'util';
import request from 'request';
import data from '../src/data.js';
import flags from './flags.js';

const headRequest = promisify(request.head);

if (process.env.CI !== 'true') {
  core.error = console.error;
  core.setFailed = console.error;
}

const schema = Joi.object({
  name: Joi.string().required(),
  description: Joi.string().required(),
  url: Joi.string()
    .uri()
    .required(),
  country: Joi.string()
    .valid(...flags)
    .required(),
  twitter: Joi.string().pattern(new RegExp(/^@?(\w){1,15}$/)),
  emoji: Joi.string().allow(''),
  computer: Joi.string().valid('apple', 'windows', 'linux'),
  phone: Joi.string().valid('iphone', 'android'),
  tags: Joi.array().items(Joi.string()),
});

(async function() {
  const validatedPersons = data.map(person => schema.validate(person));

  let errorExists = false;

  for await (const person of validatedPersons) {
    if (person.error) {
      errorExists = true;
      core.error(person.value.name);
      person.error.details.forEach(d => core.error(d.message));
    }

    const hasUrlError =
      person.error &&
      !person.error.details.map(d => d.path).some(p => p === 'url');

    if (!person.error || !hasUrlError) {
      try {
        const res = await headRequest({
          url: person.value.url,
          timeout: 3000,
        });
        if (res.statusCode !== 200) {
          core.error(`${person.value.url} has problems`);
          errorExists = true;
        }
      } catch (e) {
        core.error(`${person.value.url} ${e.message}`);
        errorExists = true;
      }
    }
  }

  if (errorExists) {
    core.setFailed('Action failed with validation errors, see logs');
  }
})();

⇒  node scripts/data-validate.js
(node:21398) ExperimentalWarning: The ESM module loader is experimental.
Wes Bos
"url" is required
https://renepot.com/uses Hostname/IP does not match certificate's altnames: Host: renepot.com. is not in the cert's altnames: DNS:renepot.net, DNS:www.renepot.net
https://tforster.com/uses ETIMEDOUT
https://hussain4real.github.io/Portfolio/uses has problems
https://scotttolinski.com/uses ETIMEDOUT
https://calypsobronte.me/uses ETIMEDOUT
https://pavel.dev/uses certificate has expired
https://sanketgandhi.com/uses has problems
https://www.richardpalaciosg.dev/uses has problems
https://blog.andrejjovanovic.com/uses has problems
https://martinchammah.dev/uses has problems
https://racik.info/uses has problems
https://kilinkis.me/uses self signed certificate
https://khriztianmoreno.dev/uses has problems
https://mynameisyuri.com/uses has problems
https://carloscharris.com/uses ETIMEDOUT
https://aendrew.com/uses self signed certificate
Action failed with validation errors, see logs

[HugoDF]

Yeah came up with something similar and have the following output

::error::URL: "https://sam-cross.github.io/uses" failed with status: 404
::error::URL: "https://hussain4real.github.io/Portfolio/uses" failed with status: 404
::error::URL: "https://jasoncoryalvernaz.com/uses" failed with status: 301
::error::URL: "https://renepot.com/uses" failed with error: Error [ERR_TLS_CERT_ALTNAME_INVALID]: Hostname/IP does not match certificate's altnames: Host: renepot.com. is not in the cert's altnames: DNS:renepot.net, DNS:www.renepot.net
::error::URL: "https://scotttolinski.com/uses" failed with status: 301
::error::URL: "https://calypsobronte.me/uses" failed with status: 301
::error::URL: "https://laurosilva.com/uses" failed with status: 301
::error::URL: "https://glennreyes.com/uses" failed with status: 301
::error::URL: "https://aless.co/uses" failed with status: 301
::error::URL: "https://bradgarropy.com/uses" failed with status: 301
::error::URL: "https://www.euperia.com/uses" failed with status: 301
::error::URL: "https://kentcdodds.com/uses" failed with status: 301
::error::URL: "https://speek.design/uses" failed with status: 301
::error::URL: "https://www.davidosomething.com/uses/" failed with status: 302
::error::URL: "https://vincentramdhanie.com/uses" failed with status: 301
::error::URL: "https://www.silvandiepen.nl/uses/" failed with status: 301
::error::URL: "https://joshuabarker.com/uses" failed with status: 301
::error::URL: "https://abdisalan.com/uses" failed with status: 301
::error::URL: "https://pavel.dev/uses" failed with error: Error: certificate has expired
::error::URL: "https://jonsuh.com/uses" failed with status: 301
::error::URL: "https://pouria.dev/uses" failed with status: 301
::error::URL: "https://sanketgandhi.com/uses" failed with status: 404
::error::URL: "https://www.richardpalaciosg.dev/uses" failed with status: 404
::error::URL: "https://gokul.site/uses" failed with status: 301
::error::URL: "https://jeremymouzin.com/uses" failed with status: 301
::error::URL: "https://rafaelquintanilha.com/about#uses" failed with status: 301
::error::URL: "https://www.arcath.net/uses" failed with status: 301
::error::URL: "https://benleivian.com/uses" failed with status: 301
::error::URL: "https://ssmusoke.com/uses" failed with status: 301
::error::URL: "https://steveheyes.co.uk/uses" failed with status: 301
::error::URL: "https://timothymiller.dev/uses" failed with status: 301
::error::URL: "https://martinchammah.dev/uses" failed with status: 404
::error::URL: "https://steven.vanloef.com/uses" failed with status: 301
::error::URL: "https://racik.info/uses" failed with status: 404
::error::URL: "https://byurhanbeyzat.com/uses" failed with status: 301
::error::URL: "https://danielvanc.com/uses" failed with status: 301
::error::URL: "https://chiamakaikeanyi.dev/uses" failed with status: 301
::error::URL: "https://terrygodier.com/uses" failed with status: 301
::error::URL: "https://kilinkis.me/uses" failed with error: Error: self signed certificate
::error::URL: "https://jamesmills.co.uk/uses" failed with status: 301
::error::URL: "https://hakaselogs.me/2020-01-10/what-i-use" failed with status: 301
::error::URL: "http://carlosjunod.me/uses/" failed with status: 302
::error::URL: "https://wesbos.com/uses" failed with status: 301
::error::URL: "https://www.mihaiserban.dev/uses" failed with status: 301
::error::URL: "https://kaleighscruggs.com/uses" failed with status: 301
::error::URL: "https://johngarrett.dev/uses" failed with status: 301
::error::URL: "https://khriztianmoreno.dev/uses" failed with status: 404
::error::URL: "https://tomhazledine.com/uses" failed with status: 301
::error::URL: "https://mynameisyuri.com/uses" failed with status: 404
::error::URL: "https://elliotbonneville.com/uses" failed with status: 301
::error::URL: "https://nickjanetakis.com/uses" failed with status: 301
::error::URL: "https://aendrew.com/uses" failed with error: Error: self signed certificate
::error::URL: "https://dominicode.com/uses" failed with status: 301
::error::URL: "https://tforster.com/uses" failed with error: Error: Request timed out
::error::URL: "https://deanacus.com/uses/" failed with error: Error: Request timed out
::error::URL: "https://carloscharris.com/uses" failed with error: Error: Request timed out
::error::Action failed with 56 URL fetch failures, see logs

[HugoDF]

@wesbos @iamandrewluca Have created 2 separate MRs with the schema validation and the actual pinging

Not sure how we want to proceed when we do find dodgy links?

[iamandrewluca]

I think that links that redirect to same url with or without slash at end should be allowed. Also a solution to not run validation on all users. Diff data.js from master to one from PR and validate only new users.

[HugoDF]

@iamandrewluca that's what I ended up with as well, redirecting your /uses to /uses/, another site or an annual post is fine as long as it shows something (not a 404 or hanging).

[HugoDF]

I've also commented out the users who /uses page doesn't work as expected cc @wesbos