search

datacite / akita

DataCite Commons
https://commons.datacite.org
MIT License
6 stars 3 forks source link

Specify deny for all machine access except google #369

Closed richardhallett closed 1 month ago

richardhallett commented 1 month ago

Purpose

Specify more general deny for machine access from crawling commons. API Access is preferred for accessing data.

The exception is Google (at least for now).

Approach

Update robots txt

Note: robots.txt is only advisory this does not necessarily stop machine access.

Open Questions and Pre-Merge TODOs

N/A

Learning

N/A

Types of changes

Reviewer, please remember our guidelines:

cypress[bot] commented 1 month ago

2 flaky tests on run #1274 ↗︎

0 55 0 0 Flakiness 2

Details:

Merge ee3456a2f0ef03ddbd35d1c8a470d6425312723a into 88e26e2defb1e2e459ca0012b2bf...
Project: akita Commit: ac8249f0d3 ℹ️
Status: Passed Duration: 03:27 💡
Started: Jul 16, 2024 11:59 AM Ended: Jul 16, 2024 12:02 PM
Flakiness  statistics.test.ts • 1 flaky test • Tests View Output
Test Artifacts
Overview > header Test Replay Screenshots
Flakiness  search.test.ts • 1 flaky test • Tests View Output
Test Artifacts
... > search with enter Test Replay Screenshots

Review all test suite changes for PR #369 ↗︎

cypress[bot] commented 1 month ago

2 flaky tests on run #1278 ↗︎

0 55 0 0 Flakiness 2

Details:

Merge pull request #369 from datacite/robots_deny
Project: akita Commit: 56b1f8abf5
Status: Passed Duration: 04:50 💡
Started: Jul 17, 2024 1:03 PM Ended: Jul 17, 2024 1:07 PM
Flakiness  search.test.ts • 1 flaky test • Tests View Output
Test Artifacts
... > search with click Test Replay Screenshots
Flakiness  fundrefContainer.test.ts • 1 flaky test • Tests View Output
Test Artifacts
FundrefContainer > not in ror Test Replay Screenshots

Review all test suite changes for PR #369 ↗︎