search

datamade / bga-payroll

💰 How much do your public officials make?
4 stars 4 forks source link

Disallow crawling of data export links #568

Closed hancush closed 2 years ago

hancush commented 2 years ago

We do not want bots to crawl the export links. Add nofollow, like we did for source files.

Because the standardized data download is relatively heavy and not cached, we may also want to restrict downloads to users who have given us an email address. Cc: @fgregg

fgregg commented 2 years ago

thanks, @hancush. @smcalilly can add the nofollow.

do you have suggestions on how to add the restriction? @hancush ?

hancush commented 2 years ago

iirc, "authenticated" users will have some cookie set. you could check for that 1) when rendering a page w/ the download button, to determine whether to show the download link and 2) in the download view?

fgregg commented 2 years ago

we would want this on rendering the download button.

we probably want a more complicated interaction here, where the download button is there, but if the cookie is not present, then clicking prompts the user to login, and if the cookie is there it is a normal link.

I'll open an issue for that behavior. To close this issue, @smcalilly, just add the nofollow link.