endless loop when 2 requests happens

[aegiacometti]

Is your feature request related to a problem? Please describe. I have a script to add hosts to the group. The script login, get the group, search in the response if the object is already in the group, if not add, if yes skip, finally verify. This involves several get/post/get operations. While executing the script, if another user tries to execute the script again, it will invalidate the first user token, and the first script will lock in an endless loop until the refresh timer.

Describe the solution you'd like If there was a successful first login, then wait a few seconds and try to log in again.

[daxm]

Hmm. I'm not sure how to address this unless we check to see if anyone else is logged into the FMC. Even at that point, how would we know what they are doing while logged in? I'm not sure I understand your suggested resolution. Can you describe in more detail?

[marksull]

Yeah we see this issue often when we have functional testing that run in parallel and they both try to run tests on the same device using the same account. The first session is invalidated as soon as the second session logs in, and there is no way to renew the token unless you login as a new session which will invalidates the second and around and around we go.

I can't think of an obvious way to solve this (other than ensuring you have multiple admin accounts to avoid concurrent connections with the same account) because how long do you wait? We have some jobs that are connected for 10 mins via the API....

[aegiacometti]

What do you exactly mean with 10 minutes jobs? Do they involve several independant requests?With independent I mean if that they can expire (because of a second session login) and relogin and continue from there.If this is the case. Maybe some exponential delay/retry algorithm could help to avoid collisions. Retry in 2, 4, 8, 16 etcEl 8 dic. 2020 16:49, Mark Sullivan notifications@github.com escribió: Yeah we see this issue often when we have functional testing that run in parallel and they both try to run tests on the same device using the same account. The first session is invalidated as soon as the second session logs in, and there is no way to renew the token unless you login as a new session which will invalidates the second and around and around we go. I can't think of an obvious way to solve this (other than ensuring you have multiple admin accounts to avoid concurrent connections with the same account) because how long do you wait? We have some jobs that are connected for 10 mins via the API....

—You are receiving this because you authored the thread.Reply to this email directly, view it on GitHub, or unsubscribe.

[aegiacometti]

For sure the userid should be used only for api requests and not for normal use.I don't think the api will logoff a user in the middle of a operation like post, cause i could cause some inconsistency.Hard to test...El 7 dic. 2020 21:45, daxm notifications@github.com escribió: Hmm. I'm not sure how to address this unless we check to see if anyone else is logged into the FMC. Even at that point, how would we know what they are doing while logged in? I'm not sure I understand your suggested resolution. Can you describe in more detail?

—You are receiving this because you authored the thread.Reply to this email directly, view it on GitHub, or unsubscribe.

[daxm]

I'll try to jump in here and answer.  The FMC server only allows 1 simultaneous login per user (hence the recommendation to create a special "apiadmin" user).  However, this recommendation makes the assumption that only 1 user at a time is accessing the FMC via that user!  So, you have have multiple people running scripts against the FMC server using the same "apiadmin" account then they can kick each other out.

Additionally, it has been my experience that the FMC will "auto kick" an existing user when a new login request comes in.  There is no feedback asking whether you really want to override the existing user's connection.  I know something like this feedback exists when you log into the FMC via the GUI but I don't know of any mechanism from the API access point of view.

However, regardless if you have multiple accounts, one for each user running scripts, the FMC will just overwrite an entry based on whomever last updated that entry.  So, for example, let's say that your script gathers info about 'Rule 1' and my script gathers info about 'Rule 1' at the same time.  Then I make changes and push them to the FMC.  Then, after that, your script does the same thing.  Your script's changes will be the ones kept.

Does this help?

On 12/8/20 8:34 AM, aegiacometti wrote:

What do you exactly mean with 10 minutes jobs? Do they involve several independant requests?With independent I mean if that they can expire (because of a second session login) and relogin and continue from there.If this is the case. Maybe some exponential delay/retry algorithm could help to avoid collisions. Retry in 2, 4, 8, 16 etcEl 8 dic. 2020 16:49, Mark Sullivan notifications@github.com escribió: Yeah we see this issue often when we have functional testing that run in parallel and they both try to run tests on the same device using the same account. The first session is invalidated as soon as the second session logs in, and there is no way to renew the token unless you login as a new session which will invalidates the second and around and around we go. I can't think of an obvious way to solve this (other than ensuring you have multiple admin accounts to avoid concurrent connections with the same account) because how long do you wait? We have some jobs that are connected for 10 mins via the API....

—You are receiving this because you authored the thread.Reply to this email directly, view it on GitHub, or unsubscribe.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/daxm/fmcapi/issues/94#issuecomment-740693174, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAZOMZ2HDBM2BTVRJSQGVQTSTZBRLANCNFSM4UQM5X5Q.

[marksull]

What do you exactly mean with 10 minutes jobs? Do they involve several independent requests?With independent I mean if that they can expire (because of a second session login) and relogin and continue from there.If this is the case.

We have jobs that deploy 10000 or more lines of rules to Policies at a time. At best the FMC Bulk API supports 1000 lines per payload which takes up to 1 minutes to be processed. As a result it is easy for a deployment to take 10+ minutes and there are literally 1000's of API calls during this window.

Maybe some exponential delay/retry algorithm could help to avoid collisions. Retry in 2, 4, 8, 16

But how do you delay/retry without affecting the last session that was logged in? If session 1 gets terminated because another script 2 logged in, if the original script tries after X period of time, they will in effect kill the session of the second script, and hence the loop I mentioned above begins.

I don't think the api will logoff a user in the middle of a operation like post, cause i could cause some inconsistency.Hard to test...

It won't terminate you mid-Post, but it will happily terminate you mid-10000 posts.

Going forward....

If you wanted change the behaviour yourself to implement your own back-off by applying a monkey patch to fmcapi.fmc.Token.get_token which currently will ONLY refresh the token if the refresh time has been exceeded (which is the problem you are seeing). But be aware, if you call the Token.generate_tokens() you will again terminate a currently operating session so this will be a case of session-killing-session-killing-ad-infinitum.

How we are handling this right now:

Because we have all our automation in containers and there is a potential that multiple containers can be spun at at the same potentially accessing the same devices with the same credentials, we are using a semaphore to control which script has access to the account. Our scripts (will) grab the semaphore when they start to perform an action and start on their merry way. When finished, they release the semaphore. If another job starts up and find the semaphore is currently being used, it goes into a holding pattern until the semaphore is released. You can implement the semaphore with a DB, a file, vault, api or any method that your scripts can communally access. Not perfect, and quite annoying to have to implement this to be 100% honest, but solves the problem without creating another problem.

[aegiacometti]

Hi, thanks a lot for taking the time to reply. And sorry, i’ve been traveling. I Will come back to this in a couple of days.

De: Mark Sullivanmailto:notifications@github.com Enviado: Wednesday, December 9, 2020 6:25 AM Para: daxm/fmcapimailto:fmcapi@noreply.github.com CC: aegiacomettimailto:aegiacometti@hotmail.com; Authormailto:author@noreply.github.com Asunto: Re: [daxm/fmcapi] endless loop when 2 requests happens (#94)

What do you exactly mean with 10 minutes jobs? Do they involve several independent requests?With independent I mean if that they can expire (because of a second session login) and relogin and continue from there.If this is the case.

We have jobs that deploy 10000 or more lines of rules to Policies at a time. At best the FMC Bulk API supports 1000 lines per payload which takes up to 1 minutes to be processed. As a result it is easy for a deployment to take 10+ minutes and there are literally 1000's of API calls during this window.

Maybe some exponential delay/retry algorithm could help to avoid collisions. Retry in 2, 4, 8, 16

But how do you delay/retry without affecting the last session that was logged in? If session 1 gets terminated because another script 2 logged in, if the original script tries after X period of time, they will in effect kill the session of the second script, and hence the loop I mentioned above begins.

I don't think the api will logoff a user in the middle of a operation like post, cause i could cause some inconsistency.Hard to test...

It won't terminate you mid-Post, but it will happily terminate you mid-10000 posts.

Going forward....

If you wanted change the behaviour yourself to implement your own back-off by applying a monkey patch to fmcapi.fmc.Token.get_token which currently will ONLY refresh the token if the refresh time has been exceeded (which is the problem you are seeing). But be aware, if you call the Token.generate_tokens() you will again terminate a currently operating session so this will be a case of session-killing-session-killing-ad-infinitum.

How we are handling this right now:

Because we have all our automation in containers and there is a potential that multiple containers can be spun at at the same potentially accessing the same devices with the same credentials, we are using a semaphore to control which script has access to the account. Our scripts (will) grab the semaphore when they start to perform an action and start on their merry way. When finished, they release the semaphore. If another job starts up and find the semaphore is currently being used, it goes into a holding pattern until the semaphore is released. You can implement the semaphore with a DB, a file, vault, api or any method that your scripts can communally access. Not perfect, and quite annoying to have to implement this to be 100% honest, but solves the problem without creating another problem.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/daxm/fmcapi/issues/94#issuecomment-741219249, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AEABLOYUCWURN7KXYTGFQSLST2YUTANCNFSM4UQM5X5Q.

[aegiacometti]

Hi just created the PR for this... please let me know what do you think

[aegiacometti]

solved in PR. Thanks! wonderful experience contributing to this project.