Closed wing5wong closed 4 years ago
Hi @wing5wong, How many entries per collection do you have? The CMS uses caching extensively and even conditional requests to avoid reaching the rate limit, but in some cases that's not enough.
As for pagination, the CMS uses the git tree API which allows listing around 100k files in a single request, but in order to get a file content we need to send a request per file. The CMS actually simulates pagination on the client (when using GitHub) and retrieves files in batches of 20 while you scroll down to avoid reading all the files.
The CMS doesn't paginate when the user sorts the collection since it needs to retrieve all files, read and parse their content in order to perform the sort. When the files are not cached and there are many files this can cause the limit to be reached. This is mainly an issue for new users that don't have the content cached.
When the CMS detects the limit it reached, it throttles the requests, but in the case of GitHub that doesn't help much since the limit is reset once per hour.
Possible solutions:
git-gateway
) if possible. When using git-gateway
a single account is used to connect to the repo, thus the limit is shared across different users. When using GitHub directly, each user gets a dedicated limit (and doesn't block others when reached). Downside - users need a GitHub account.sortableFields
to an empty array to disable sorting: https://www.netlifycms.org/docs/configuration-options/#sortablefieldsI would go with option 1, if possible.
Hey there thanks for the feedback. Actually i found one other large collection, our news. overall not many items given the numbers you stated right?
News: 2515 Assessments: 799 Courses: 214
will be costly for us at this point which is one of the limiting factors.
can you clarify here? Do you mean for example have a "SubjectSetup" type file with a list widget? Otherwise im not sure a file relation is suitable, as they files may not already exist (sometimes the user adds additional courses or assessments)
ideally we keep this so il avoid this for now, but keep it in mind
News: 2515 Assessments: 799 Courses: 214
The stated limit is 5000 requests per hour, so a new user sorting some collections can break the limit easily I think (we also send requests to get the commit date and some other metadata). Note that the limit resets only once an hour.
2. can you clarify here? Do you mean for example have a "SubjectSetup" type file with a list widget? Otherwise im not sure a file relation is suitable, as they files may not already exist (sometimes the user adds additional courses or assessments)
Sure, it is a relatively new feature for targeting lists in files: The target collection can be (this is a slimmed version):
- name: "subject_areas"
label: "Subject Areas"
files:
- name: areas
file: source/subject_areas.json
format: json
fields:
- label: "Areas"
widget: "list"
name: "areas"
fields:
- { label: "Title", name: "title", widget: "string" }
Then you can configure the relation via:
- label: "Related Subjects"
name: subject_areas
widget: "relation"
collection: "subject_areas"
file: "areas"
searchFields: ["areas.*.title"]
valueField: "areas.*.title"
This was the CMS would only need to read a single file to resolve the relation.
See more here: https://www.netlifycms.org/docs/widgets/#relation
Closing this as I don't see what else can be done on the CMS side. Please comment if still relevant
No worries. Thought about a few of these and have just resorted back to string/list widgets for these fields.
Using Jigsaw Static site generator i require to have these single files (i believe) in order to generate the required data for my pages.
i seem to get ~800 requests sent when typing in relation fields seems to reach the rate-limit thresholds extremely fast ( e.g. typing "DTA2" in courses relation)
Thanks for the info regardless!
On 5/08/2020, at 9:56 PM, Erez Rokah notifications@github.com wrote:
Closed #4097https://github.com/netlify/netlify-cms/issues/4097.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/netlify/netlify-cms/issues/4097#event-3623926472, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABBT5UMJ7ZZM5QWKL3NHGELR7EUDHANCNFSM4PPLWTQQ.
Once those requests are cached it shouldn't impact the rate limit thought (you'd still see them as requests in the browser, but they will be retrieved from the browser cache or get a 304 Not Modified
response which doesn't count under the limit).
@erezrokah hi there, just feeding back into this some more - today our user has made 11 commits (commits labeled "2 hours ago" ) and has since been locked out. She is the only user accessing the site currently.
another hour has passed and 4 commits made - locked out by rate limiting.
looking at our config file (below) for NEWS (2548 items) , ANNOUNCEMENTS (1 item) and NEWSLETTERS (39 items) below, I have disabled sorting on the NEWS collection. all that remains there are some relation widgets within the news collection
Im absolutely lost as to what can be causing this now - the only thing i can think of is to remove the relation fields from the NEWS collection... is that even going to help?
Looking in Devtools > Application>Storage>IndexedDB there exists approx 9500 entries already. This is around double the actual files in the folder - is my understanding correct that as these are cached, they should NOT be contributing to the rate limiting?
Is her cache somehow invalid and all these items are being re-cached each time? Something just doesn't add up now! Assuming she already has all these entries cached on her machine, we should b seeing a total of 15 (11 + 4) API requests to github, correct? even at 100x that number, we should be well below the 5000 limit.
- name: "news"
label: "News"
label_singular: "News"
folder: "source/_news"
create: true
sortableFields: []
slug: "{{year}}-{{month}}-{{day}}_{{slug}}"
summary: "{{year}}-{{month}}-{{day}} | {{title}}"
fields:
- { label: "Title", name: "title", widget: "string" }
- label: "Feature Image"
name: "feature_image"
widget: "object"
required: false
fields:
- { label: "Image Filename", name: "image", widget: "image", required: false }
- { label: "Image Description", name: "description", widget: "string", required: false }
- { label: "Image Alt Text", name: "alt", widget: "string", required: false }
- { label: "Image Credit", name: "credit", widget: "string", required: false }
- { label: "Short Intro", name: "short", widget: "text", required: false, hint: "Use this to override the default generated option which is the first 200 characters of the body" }
- { label: "Body", name: "body", widget: "markdown" }
- label: "News Author"
name: "news_author"
widget: "object"
required: false
fields:
- { label: "Date", name: "date", widget: "datetime", required: false }
- { label: "Author", name: "name", widget: "string", required: false }
- { label: "Publication", name: "publication", widget: "string", required: false }
- label: "Image Gallery"
name: "image_gallery"
widget: "list"
fields:
- { label: "Image Filename", name: "image", widget: "image" }
- { label: "Image Description", name: "description", widget: "string", required: false }
- { label: "Description", name: "description", widget: "hidden", required: false }
- { label: "Related Departments", name: faculties, required: false, multiple: true, widget: "relation", collection: "faculties", searchFields: ["title"], valueField: "title", displayFields: ["title"] }
- { label: "Related Subjects", name: subject_areas, required: false, multiple: true, widget: "relation", collection: "subject_areas", searchFields: ["title"], valueField: "title", displayFields: ["title"] }
- { label: "Related Extracurricular Areas", name: extracurricular_areas, required: false, multiple: true, widget: "relation", collection: "extracurricular_areas", searchFields: ["title"], valueField: "title", displayFields: ["title"] }
- { label: "Related Extracurricular Activities", name: extracurricular_activities, required: false, multiple: true, widget: "relation", collection: "extracurricular_activities", searchFields: ["title"], valueField: "title", displayFields: ["title"] }
- { label: "Tags", name: "tags", widget: "list", required: false }
- { label: "Show in News Sliders", name: "show_in_slider", widget: "boolean", default: true, required: false }
- { label: "Date", name: "date", widget: "datetime" }
- { label: "Image Filename", name: "image", widget: "hidden", required: false }
- { label: "Image Title", name: "image_title", widget: "hidden", required: false }
- { label: "Image Alt", name: "image_alt", widget: "hidden", required: false }
- { label: "Image credit", name: "image_credit", widget: "hidden", required: false, hint: "e.g. Andy McGechan" }
- name: "newsletters"
label: "Newsletters"
label_singular: "Newsletter"
folder: "source/_newsletters"
create: true
slug: "{{year}}-{{month}}-{{day}}_{{slug}}"
summary: "{{month}} {{year}} | {{title}}"
fields:
- { label: "Title", name: "title", widget: "string" }
- { label: "Date", name: "date", widget: "datetime" }
- { label: "Tags", name: "tags", widget: "list", required: false }
- { label: "Filename", name: "file", widget: "file", allow_multiple: false }
- { label: "Body", name: "body", widget: "markdown", required: false }
- name: announcements
label: Announcements
label_singular: "Announcement"
folder: "source/_announcements"
create: true
slug: "{{year}}-{{month}}-{{day}}_{{slug}}"
fields:
- { label: "Title", name: "title", widget: "string", hint: 'Student Achievement Conferences' }
- { label: "Emergency", name: "is_emergency", widget: "boolean", default: false }
- { label: "Date", name: "date", widget: "datetime" }
- { label: "Subtitle", name: "subtitle", widget: "string", required: false, hint: 'Students released at 1.50pm (end of Period 4)' }
- { label: "Body", name: "body", widget: "markdown" }
- { label: "Image Filename", name: "image", widget: "image", required: false }
- { label: "Tags", name: "tags", widget: "list", required: false }
@wing5wong, sorry to hear you're still experiencing issues.
The limit is tied not to a specific user of the CMS, but to the account token configured under git-gateway
settings in Netlify. Creating a separate GitHub account dedicated to the CMS might help to diagnose as we can verify that the only consumer of the token (hence contributing to reaching the limit) is the CMS.
This is around double the actual files in the folder
Seeing double the amount in cache is expected as we cache both the file content, and some meta data for the file (when it was last accessed and the author).
It would be helpful to share the network traffic from the user, so we can see which requests are sent to the API.
When capturing the traffic, make sure the Disable Cache
option is not checked.
The limit is tied not to a specific user of the CMS, but to the account token configured under
git-gateway
settings in Netlify. Creating a separate GitHub account dedicated to the CMS might help to diagnose as we can verify that the only consumer of the token (hence contributing to reaching the limit) is the CMS.
Does this require adding paid users to my team or can I create the account with access to my repo and just use an access token generated directly on github? (independent of my netlify sign in through github - clicking "Generate access token in GitHub" appears to directly use my current credentials to generate this token)
Seeing double the amount in cache is expected as we cache both the file content, and some meta data for the file (when it was last accessed and the author).
OK, that's what I thought was the case
It would be helpful to share the network traffic from the user, so we can see which requests are sent to the API. When capturing the traffic, make sure the Disable Cache option is not checked.
I will try get this done today depending on availability.
Does this require adding paid users to my team or can I create the account with access to my repo and just use an access token generated directly on github?
Any account with write access to the repo should work by generating the token from https://github.com/settings/tokens/new
Hi @erezrokah Sorry for delays here. as usual life gets in the way.
I havent tried the new token just yet, but as a point of reference I was with our maintainer yesterday after she had been rate limited earlier in the day - then from a fresh start, she tried to edit an article.
It asked her if she wanted to restore a local backup (presumably because she was rate limited previously, and closed the page)
Then there are requests for 2500+ items
Then we refreshed the page.
Then there is a similar number of items, before hitting the rate limit and seeing the dreaded red request/responses (not visible in this image)
Is the CMS meant to reload all items when editing an article? Is there a way to tell if these items are retrieved from cache? I have just reproduced this issue myself this morning following the same process - now rate limited for an hour.
Our news folder currently:
Re-opening this so we can track this better
Thanks.
I have created a new users and invited them as a collaborator to this repository. I have generated an access token for that user and assigned it in the netlify git-gateway settings. I have tested some updates to ensure our content manager can still post updates.
Will advise if any further timeouts
Open admin page > click News category
Select Edit 1 new item
I refreshed the edit screen, and unfortunately chrome dev tools crashed.
Closing and reopening devtools reveals
still ongoing issues around this -editor today has made some posts 5 hours ago, then when trying to create a new article is rate limited before even submitting the article.
What else can be changed regarding this? we have already utilised the new API key. This is the same user every time using the site, so content should be cached correctly, meaning we don't run into this rate limiting beyond her first usage warming up the cache data
We continue to have this issue despite recreating tokens several times now. Recreating the token allows access for a short time, then we receive API timeouts again.
Something must be going on here and either cached data is not retrieved correctly, or hits against cached data are still counted against the API limit.
What other explanation can there be for this behaviour @erezrokah ?
Just yesterday i recreated an access token, our content manager could continue working and this morning after making just a couple of file update, she is rate limited This is slowly whittling away at our content managers sanity!
Sorry for the lack of response here @wing5wong. Trying to map out places we are reading many entries:
Maybe that will help us focus on what's triggering the limit.
Something must be going on here and either cached data is not retrieved correctly, or hits against cached data are still counted against the API limit.
You could observe caching behaviour using the browser network tab.
Blobs should always be retrieved from the cache after initial load. We are forcing those here https://github.com/netlify/netlify-cms/blob/519cb2d4c2db729d2643c9116f93656b6a9dba23/packages/netlify-cms-backend-github/src/API.ts#L643
Sorry for the lack of response here @wing5wong. Trying to map out places we are reading many entries:
No worries, i appreciate i am just one person and so are you. happy to continue working to identify the issue here as time permits.
- Sorting/Filtering
We've disabled sorting since we first encountered this so i think this is unlikely
- i18n
I don't believe we are using this unless it is applied automatically
- Nested collections
Again i dont think we are using this feature.
- Relation widgets
Our news collection has 4 relation widgets, so this is a possibility
- First load of editorial workflow (will load all drafts in the background) - see #3468
We are not using editorial workflow
You could observe caching behaviour using the browser network tab.
Blobs should always be retrieved from the cache after initial load. We are forcing those here
It really seems like this is not happening. I opened our events page which has only a small number of items ( so i dont get locked out!), then closed the browser. Reopening the browser and navigating to the events collection again, as well as refreshing the page, shows the data downloaded again
Thanks @wing5wong. This is very strange indeed. Maybe we're missing some basics here:
netlify-cms@2.10.67
or netlify-cms-app@2.13.2
).github
backend (like I suggested) just for the sake of debugging this (seeing if the requests to blobs are forced cached).Hi @erezrokah
Thanks for that, i have started testing those with some interesting results.
- Making sure the CMS version is the latest (
netlify-cms@2.10.67
ornetlify-cms-app@2.13.2
).
i use https://unpkg.com/netlify-cms@^2.0.0/dist/netlify-cms.js
as my script url, which at this stage is linking to https://unpkg.com/netlify-cms@2.10.67/dist/netlify-cms.js
- Trying with a different browser (I know this has been happening to different users so chances are slim here, but just in case).
Now heres where it gets interesting.... Firefox successfully returns cached data!
- Trying from a different network connection.
Il be able to try this tonight from home
- Creating a new Netlify site just for the sake of debugging this.
- Using
github
backend (like I suggested) just for the sake of debugging this (seeing if the requests to blobs are forced cached).
I can test these but as these would be most disruptive at this stage id prefer to wait.
With the Are we able to confirm an issue with chrome caching at all? I do notice if i switch between work and personal chrome accounts that the "initiator" in the work account is "unsentRequest.js:41" and in my personal account is "VM484:1" . regardless, both of these do not appear to load a cached response.
Now heres where it gets interesting.... Firefox successfully returns cached data!
This is great news. At least we have a path forward.
I think one can control the caching behaviour on Chrome using CLI flags.
Can you verify neither one of --disk-cache-dir,--disk-cache-size,--media-cache-size
is set when launching Chrome (if they are set it will probably be reflected in the Windows shortcut).
I think one can control the caching behaviour on Chrome using CLI flags. Can you verify neither one of
--disk-cache-dir,--disk-cache-size,--media-cache-size
is set when launching Chrome (if they are set it will probably be reflected in the Windows shortcut).
Ive checked everywhere i can think regarding this - Shortcut, chrome flags, gsuite admin - either no mention of the flag, or the settings remain default (unconfigured) - except the disabled AppCache (this is disabled by default by gsuite). I dont think the caching is configured to use this though, and using indexedDB rather. Perhaps this is it?
Have tested this on a new network now and it seems the caching is functioning as expected here.
My assumption now is that somewhere along the line our work ISP is affecting the caching
Have tested this on a new network now and it seems the caching is functioning as expected here.
My assumption now is that somewhere along the line our work ISP is affecting the caching
This is very odd, especially considering it works with Firefox. Thank you for the update.
The only mechanism I can think of that could cause this is the work network injecting scripts into the traffic changing the browser fetch
implementation.
I'm not sure what else can be done on the CMS side
Closing this issue per my last comment
Thanks.
I appreciate your time in this and still haven’t managed to track exactly what is causing this but agree there nothing else we can do.
Regards, Sean Anderson
On 26/11/2020, at 11:16 AM, Erez Rokah notifications@github.com wrote:
Closing this issue per my last comment
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/netlify/netlify-cms/issues/4097#issuecomment-733969815, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABBT5UJ7DZAO2ASRUFU5EGDSRV63JANCNFSM4PPLWTQQ.
Issue: Our admin users are currently trying to update information relating to courses, assessment and subject areas. Many time the Relation widget is extremely slow to load, and often searching for items takes an incredible time, or hits rate limits. i.e. accessing a collection from the menu shows ' Loading Entries -> Caching Entries ->This may take several minutes' and never loads.
The user logs in. Clicks "NCEA Standards" Tries to modify the courses relation.
The user tries to find an assessment by opening the "NCEA Standards" collection. Infinite scroll towards the right assessment, more records load, repeat. Eventually can not find the standard quickly, so try searching it. Now the search loads forever.
Im not entirely sure this is a bug within CMS or just our setup is not quite the right fit for this CMS and causing issues.
Expected behaviour: The relation widgets should load quickly. (<3s) The infinite scroll causes issues with rate limiting seemingly. it would be easier to load all the items (or increase the per page count perhaps)
Applicable Versions:
CMS configuration