Investigate limiting number of datasets that can be created in a time period

[cjhendrix]

To avoid DOS.

[teodorescuserban]

Given I have a list of URLs used to create datasets / resources, I can limit the access to those to a certain threshold globally or per ip, e.g no more than 300 per minute globally, no more than 5 per minute per IP. This will be done in nginx config.

[teodorescuserban]

Let's make two lists:

1. for UI calls - with some limits
2. for API calls - with some other limits

[cjhendrix]

How about:

UI: 5 per minute per IP. URL https://data.hdx.rwlabs.org/dataset/new?organization_id* API: 100 per day per IP: URLs

• https://data.hdx.rwlabs.org/api/3/action/package_create*
• https://data.hdx.rwlabs.org/api/action/package_create*
• https://data.hdx.rwlabs.org/api/3/action/resource_create*
• https://data.hdx.rwlabs.org/api/action/resource_create*

tagging @mbellotti @alexandru-m-g if they might have other API or interface url ideas.

[teodorescuserban]

I am still waiting a final decision on this. In the mean time, on Monday, I will make all the setups required so only the numbers would need changing.

[teodorescuserban]

It turns out that nginx can only limit number of connections per second or per minute, which completely changes our option range... One possible solution is to implement this one on ckan level, possibly per user. tagging @cjhendrix and @alexandru-m-g

[danmihaila]

moving this to Sprint 43.

[cjhendrix]

I think per minute or per second doesn't really solve the problem for APIs. I suggest we do it in CKAN somehow. I'll move this one to Sprint 44 and we can reassign it to one of the ckan devs on the SPAR call.

[teodorescuserban]

Please reassign. :)

[cjhendrix]

50 req per minute.

Applies to anything that can create stuff in the database: datasets/resources/gallery items

[teodorescuserban]

tagging @mbellotti @danmihaila @aalecs @alexandru-m-g just to comment here what other api calls we have (beside the ones mentioned above by @cjhendrix ) that create stuff in our ckan.

[teodorescuserban]

tagging @mbellotti @danmihaila @aalecs @alexandru-m-g just to comment here what other api calls we have (beside the ones mentioned above by @cjhendrix ) that create stuff in our ckan.

I do need that list.

[cjhendrix]

Everyone ( @mbellotti @danmihaila @aalecs @alexandru-m-g ) please, think about this for two minutes and either post that you have nothing to add or post what you come up with.

[danmihaila]

I checked the CKAN API documentation and I found the following full list:

https://data.hdx.rwlabs.org/api/3/action/package_create*
https://data.hdx.rwlabs.org/api/action/package_create*
https://data.hdx.rwlabs.org/api/3/action/resource_create*
https://data.hdx.rwlabs.org/api/action/resource_create*
https://data.hdx.rwlabs.org/api/3/action/related_create
https://data.hdx.rwlabs.org/api/3/action/package_relationship_create
https://data.hdx.rwlabs.org/api/3/action/member_create
https://data.hdx.rwlabs.org/api/3/action/group_create
https://data.hdx.rwlabs.org/api/3/action/organization_create
https://data.hdx.rwlabs.org/api/3/action/rating_create
https://data.hdx.rwlabs.org/api/3/action/user_create
https://data.hdx.rwlabs.org/api/3/action/vocabulary_create
https://data.hdx.rwlabs.org/api/3/action/activity_create
https://data.hdx.rwlabs.org/api/3/action/tag_create
https://data.hdx.rwlabs.org/api/3/action/group_member_create
https://data.hdx.rwlabs.org/api/3/action/organization_member_create    
https://data.hdx.rwlabs.org/api/action/related_create
https://data.hdx.rwlabs.org/api/action/package_relationship_create
https://data.hdx.rwlabs.org/api/action/member_create
https://data.hdx.rwlabs.org/api/action/group_create
https://data.hdx.rwlabs.org/api/action/organization_create
https://data.hdx.rwlabs.org/api/action/rating_create
https://data.hdx.rwlabs.org/api/action/user_create
https://data.hdx.rwlabs.org/api/action/vocabulary_create
https://data.hdx.rwlabs.org/api/action/activity_create
https://data.hdx.rwlabs.org/api/action/tag_create
https://data.hdx.rwlabs.org/api/action/group_member_create
https://data.hdx.rwlabs.org/api/action/organization_member_create

Should we also care about delete action? Tagging @cjhendrix

[cjhendrix]

Thanks for taking a look at it Dan.

I propose we only bother with create actions on packages, resources, groups, organizations, and gallery (which doesn't seem to be in the list above). Those are the things that would be visible on the site if some made a mess with a script.

[danmihaila]

I will also add tag create and vocabulary which could mean something also...

[teodorescuserban]

ok. I will enable limits until Monday. Leave it open. thank you, @danmihaila

[danmihaila]

:+1:

[cjhendrix]

Where are we with this?

[danmihaila]

@teodorescuserban can you please give an update on this?