search

lanrat / czds

simple golang API and tools to interact with czds.icann.org
https://pkg.go.dev/github.com/lanrat/czds
GNU General Public License v3.0
72 stars 14 forks source link

-extend-all seems failling to "extend all possible zones" #14

Closed uggyuggy closed 2 years ago

uggyuggy commented 2 years ago

Hi,

tested with 1.2.6 (btw it may be useful to have some -version option)

real 0m2,720s user 0m0,081s sys 0m0,037s


- neustar remains `Extensible: true` and `ExtensionInProcess: false`after previous command.

$ ./czds-status -username "${CZDS_LOGIN}" -password "${CZDS_PASS}" -zone neustar ID: xxxx TLD: neustar (neustar) Status: approved Created: xxxx Updated: xxxx Expires: Wed Sep 28 23:59:59 2022 AutoRenew: false Extensible: true ExtensionInProcess: false ...

lanrat commented 2 years ago

Thanks for the bug report.

Some parts of the CZDS API are slow and even when successful, may take 5-10 minutes to have an affect. It's a limitation on the CZDS end.

Can you verify the renewal status by logging into the CZDS website and checking there or run czds-status again?

uggyuggy commented 2 years ago

Hi,

Now (1 hour later ?), still ExtensionInProcess: false

./czds-status -username "${CZDS_LOGIN}" -password "${CZDS_PASS}" -zone neustar
ID: xxx
TLD:    neustar (neustar)
Status: approved
Created:    xxxx
Updated:    xxxx
Expires:    Wed Sep 28 23:59:59 2022
AutoRenew:  false
Extensible: true
ExtensionInProcess: false

Web admin still display the "request extension" button too. I will try again tomorow and will update there the result.

lanrat commented 2 years ago

I've never seen it take more than 10 minutes to show the status change, so this might be a bug. I'll try to reproduce with my account.

lanrat commented 2 years ago

I was able to submit an extension for my zones.

./czds-status -username "${CZDS_LOGIN}" -password "${CZDS_PASS}" -zone mtr
ID:    ********
TLD:    mtr (mtr)
Status: approved
Created:        Thu Jun 30 22:08:42 2022
Updated:        Fri Jul  1 02:10:52 2022
Expires:        Thu Sep 29 23:59:59 2022
AutoRenew:      false
Extensible:     false
ExtensionInProcess:     true
Cancellable:    false
Request IP:      ********
FTP IPs:         [ ********]
Reason: ********
History:
        Thu Jun 30 22:08:42 2022        Request submitted
        Thu Jun 30 22:13:09 2022        Request status change to Pending
        Fri Jul  1 02:10:52 2022        Request status change to Approved
        Tue Aug 30 06:00:43 2022        Access extension requested to the Registry by the user

Perhaps this may be related to #11 where the wrong zoneID may be used? Have you been able to verify that the ID of the zone request you are getting the status of is the same as the one extended?

lanrat commented 2 years ago

When using -extend-all, are any of the zones extended, or do they all appear to have the same behavior as neustar?

uggyuggy commented 2 years ago

Thank's for trying to replicate on your side.

Another test of getting status after few hours (not tried again for now to "extend-all" again)

ID: ***
TLD:    neustar (neustar)
Status: approved
Created:    ***
Updated:    ***
Expires:    Wed Sep 28 23:59:59 2022
AutoRenew:  false
Extensible: true
ExtensionInProcess: false

So seems not related to the 5/10min delay.

Have you been able to verify that the ID of the zone request you are getting the status of is the same as the one extended?

Can you guide me on how I can see the ID of the ones that have been extended with -extend-all? Even with -verbose I don't see information if some have been extended. Does -extend-all is supposed to show that ? In my case it was showing nothing (does it means it extended nothing ?) If it is supposed to report the extended TLDs in the output of the command, seems it did not detected some were Extensible.

Note: the -extend-all complete in around 2 seconds. Is that supposed to be so quick ? (if yes, good :) )

When using -extend-all, are any of the zones extended, or do they all appear to have the same behavior as neustar?

I also have yandex and nyc expiring the same date/time as neustar which are still now Extensible: true after the previous -extend-all earlier today. (Found by looking into "Expire Date" of the report)

So from the output of the -extend-all and from few samples below, it looks none has been extended yet.

$ ./czds-status -username "${CZDS_LOGIN}" -password "${CZDS_PASS}" -zone yandex
ID: ***
TLD:    yandex (yandex)
Status: approved
Created:    ***
Updated:    ***
Expires:    Wed Sep 28 23:59:59 2022
AutoRenew:  false
Extensible: true
ExtensionInProcess: false
...

$ ./czds-status -username "${CZDS_LOGIN}" -password "${CZDS_PASS}" -zone nyc
ID: ***
TLD:    nyc (nyc)
Status: approved
Created:    ***
Updated:    ***
Expires:    Wed Sep 28 23:59:59 2022
AutoRenew:  false
Extensible: true
ExtensionInProcess: false
...

Note: from tests on previous days, the -extend worked just fine since your last code update. So the problem may be on getting the list of TLDs that are "Extensible". Is there an API curl call I can test to simulate getting this information ?

If there is some other tests I can do on my side, feel free to ask. Thank's !

lanrat commented 2 years ago

Can you guide me on how I can see the ID of the ones that have been extended with -extend-all?

Since this repo is primary an API but includes some commands to make use of it, and the extend-all functionality is all inside the API portion, there no good way to make the library print debugging statements including the IDs without breaking the compatibility for any other programs that use it.

That being said, if you are familiar with go programming, you can easily add a print statement here and compile a version to see what gets requested.

Note: the -extend-all complete in around 2 seconds. Is that supposed to be so quick ? (if yes, good :) )

It will depend on how many zones there are that can be extended. The more zones to extend the longer it will take.

Does extending work if you manually request a specific zone to be extended? For example:

$ ./czds-request -username "${CZDS_LOGIN}" -password "${CZDS_PASS}" -extend yandex

Thanks for offering to help debug this!

lanrat commented 2 years ago

When using extend-all are any zones printed as a result? All zones names that are able to be extended should print when it's done.

uggyuggy commented 2 years ago

Hi,

That being said, if you are familiar with go programming, you can easily add a print statement and compile a version to see what gets requested.

I'm not familiar with, but I will try my best :)

The more zones to extend the longer it will take.

Ok thanks.

Does extending work if you manually request a specific zone to be extended?

Yes it always work (afaik)

  1. Status Extensible: true and ExtensionInProcess: false
  2. -extend yandex 
    ./czds-request -username "${CZDS_LOGIN}" -password "${CZDS_PASS}" -extend yandex
Extended: [yandex]
  3. Status Extensible: false / ExtensionInProcess: true

When using extend-all are any zones printed as a result? All zones names that are able to be extended should print when it's done.

Yesterday, nothing was printed as result (despite some were flagged as Extensible: true

Something hot today ! I got the hot TLD that looks to have been extended (only this one) 

./czds-request -username "${CZDS_LOGIN}" -password "${CZDS_PASS}" -extend-all -verbose
2022/08/31 10:16:57 Authenticating to https://account-api.icann.org/api/authenticate
2022/08/31 10:16:59 Requesting extension for all TLDs
Extended: [hot]

./czds-status -username "${CZDS_LOGIN}" -password "${CZDS_PASS}" -zone hot
ID: ***
TLD:    hot (hot)
Status: approved
Created:    ***
Updated:    ***
Expires:    Sat Sep  3 23:59:59 2022
AutoRenew:  false
Extensible: false
ExtensionInProcess: true

hot was one that is expiring in 3 days.

Note that neustar (expiring more than in 3 days) is still now Extensible: true with Expires: Wed Sep 28 23:59:59 2022

So one question coming to my mind was "does -extend-all, extend all those that are Extensible, or does it extend all that are Extensible AND that are expiring within the next xx days (less than ~ 30) "

But according to you previous test, seems you were able to extend mtr on 30 Aug with an Expires: Thu Sep 29 23:59:59 2022 I guess this has been extended with -extend-all ? So I'm a bit lost :)

uggyuggy commented 2 years ago

Does -extend-all stop to check for further TLDs status as soon as it find one that is "non extensible" ?

I'm going to check further on this as it sounds to me that this may be the problem..

uggyuggy commented 2 years ago

Does the logic is to retrieve a list of "Approved" status, ordered with domains that expire the sooner, and stop to look further as soon as one is "non extensible" ?

In my case I suspect to have currently:

Does -extend-all reach .TLD2 to try to extend it ?

Note: my .hot TLD that magically been extended today but not yesterday, was one "impacted" by #11 (CZDS API reply first hotmail when searching for hot)

lanrat commented 2 years ago

I was just reviewing the code and I think I just spotted the bug, which is exactly what you suspected in #comment-1233320975.

Unfortunately there is no API call to get all extendable requests, so I instead I request all already approved requests and sort by their expiration date ascending. While iterating over all of these, every one that is extendable is extended, and on the first one I find that is not extendable I break the loop.

The problem is that if you have 5 extendable zones, and say the 3rd is manually extended, this logic will only extend the first two, leaving zone 4 and 5 un-extended. I'm working on an update now to not break early which should solve this issue.

lanrat commented 2 years ago

@uggyuggy can you give the latest release v1.2.7 a try?

I think this should be fixed with 51e8449624a5b50224a9d5b5e2306431ead9fe81.

uggyuggy commented 2 years ago

Hi,

Thank's for the checks and the fast modifications ! 👍

Tested with v1.2.7:

Does v1.2.7 check if a TLD is "Extensible" only for TLDs that are expiring in next 30 days (I think that is the range provided by ICANN) ? If not already that way, it may be possible to "quickly" get the expiration date for all TLDs with the report feature, then filter those "approved" and identifying those expiring in less than 30days. Which in average could be around ~ 20 TLDs ? (21 currently for me today) By checking the "Extensible" status for only ~ 20 TLDs (average), it may possibly reduce the time for -extend-all to complete ?

I made a quick POC (not in go sorry) using that way to proceed, and it looks faster than 7minutes.

Thank's again for sharing this software with the community ! 👍

lanrat commented 2 years ago

I'm glad to hear this is resolved for you!

The change I made will iterate over all currently approved zones in batches of 100 and extend each one that can be extended. Even if there are no zones to extend, each approved zone is still checked. So, this will take longer than the prior method, but be more correct.

However, 7 minutes is unexpectedly long. On my system it takes just over 1 minute to check every zone and extend as needed. I think something else is going on here.

I've intentionally chosen to not look at the expiration date for zones when determining to expend and instead look at the API response field Extensible. Right now ICANN is allowing extensions in the 30 day window, but nothing is preventing them from increasing or decreasing that again in the future, and I don't want to have to hard-code values like that into the logic.

Additionally, even if I did attempt to extend every zone within that window, the Extensible and ExtensionInProcess fields should still be checked for every zone which would require another API request for each zone likely making this take longer than checking all zones in batches of 100.

it may be useful to have some -version option :)

I agree. This is something I have been looking into, but have not gotten around to yet, but I just filed #15 to track it.

Since this issue is solved, I'll close it for now. but feel free to continue to comment here or open a new issue if you find something else.

uggyuggy commented 2 years ago

Since this issue is solved, I'll close it for now.

Sure, thank's again 👍

but feel free to continue to comment here

Thx :) So I will :)

However, 7 minutes is unexpectedly long. On my system it takes just over 1 minute to check every zone and extend as needed. I think something else is going on here.

1 min for you, mmm.. thank's for the info, as it confirm there is something to possibly investigate further on my side. I have an average of 1200 TLDs "Approved" anytime. Does this means the code attempt to extend those 1200, and all the 1200 attempts/success complete in around 1min for you (and 7min for me) ? Do you have less TLDs "Approved", so less attemps to extend, and faster ? Or you also have around 1200 ?

I tested again today, and still for me for now-> 7m42s

Right now ICANN is allowing extensions in the 30 day window, but nothing is preventing them from increasing or decreasing that again in the future, and I don't want to have to hard-code values like that into the logic.

Than'ks for the detailed answer. I agree with you, hardcoding a value may not be good. If that would improve the performances (to be confirmed first if this may have an impact on the speed), may I suggest to have this value as an optional user option like --extend-time=30 so it would attempt to extend each day only those expiring in less than XX days, which is, with the current ICANN setting, an average of ~ 20 TLDs instead of 1200 (if I understood correclty the logic).

Thx !

lanrat commented 2 years ago

The account I have been testing with currently has ~1160 zones approved.

Since I do batches of 100 (its the same as clicking the next page on the CZDS web interface), it will make 12 HTTP requests to get all of the currently approved zones statuses, which includes the Extensible and ExtensionInProcess fields. Then for each zone request that can be extended, I issue another HTTP request to make the extension.

So if you have 5 zones that can be extended, this process should only take 12+5 = 17 requests, which should not take long at all. (plus 1-2 requests at the very start for authentication)

I'm not opposed to adding an additional --extend-time flag, but it would require another endpoint added to the API, which is currently a low priority for me, but I would accept a PR.

Even with an --extend-time option, it would only reduce a few of the "next page" requests at most.

In the past, the CZDS site would become very slow for a few hours a day, probably for maintenance or something. I haven't seen it do that in a while, but perhaps you just happen to be trying during this window? I'm out of ideas and this is my best guess other than general network congestion.

PS: I'm thinking of good ways to add verbose logging inside the API cleanly without breaking anything which may help figure out what's taking so long for you.

lanrat commented 2 years ago

I just published v1.2.9 with commit 39013a2c4284ff0373af571f611a639f2d87fa4a which extends the verbose logging into the API calls.

If you run the commands with the verbose flag now, they should print everything that is going on with timestamps. That should tell you exactly what's taking so long.

Please report beck what you find. :)

uggyuggy commented 2 years ago

Thank you very much for adding logs verbosity.

It took again 7m33s to complete (0 TLD to be extended today) but very useful to have logs now.

it will make 12 HTTP requests to get all of the currently approved zones statuses

Ok. I see page 0 to page 12 in logs.

$ rg "GetRequests filter" toto | wc -l
13
$

which includes the Extensible and ExtensionInProcess fields.

I see in log GetRequests filter: &{Status:Approved Filter: Pagination:{Size:100 Page:0} Sort:{Field:expired Direction:asc}}

wich looks like what I can sent with curl with 

'{"status":"Approved","filter":null,"pagination":{"size":100,"page":0},"sort":{"field":"expired","direction":"asc"}}'

Sending this (with curl) send me back 100 ID (x12) with each like:

    {
      "requestId": "***-***-***-***",
      "tld": "boats",
      "ulable": "boats",
      "status": "Approved",
      "created": "***",
      "last_updated": "***",
      "expired": "***",
      "sftp": false
    }

With this POST request, I'm not able to see any Extensible and ExtensionInProcess fields in the response.

Then in logs, I see 1155 of those GET to each of the 1155 IDs, over the 7 minutes (12:03:48 -> 12:11:18)

$ rg "HTTP API Request: GET" toto | wc -l
1155
$ rg "HTTP API Request: GET" toto | head -1
2022/09/03 12:03:48 HTTP API Request: GET "https://czds-api.icann.org/czds/requests/***"
$ rg "HTTP API Request: GET" toto | tail -1
2022/09/03 12:11:18 HTTP API Request: GET "https://czds-api.icann.org/czds/requests/***"
$

Then, for each of those 1155 GET, only there I can see the "extensible" and "extensionInProcess"

This is those 1155 reqests that takes a long time.

Then for each zone request that can be extended, I issue another HTTP request to make the extension.

Ok, that part makes sense.

I'm a bit confused now on:

Thank's again for the open discussion !

lanrat commented 2 years ago

I made a mistake in my prior comment, I was using czds-status to test the requests instead of actually running request -extend-all because I did not have any currently available zone requests I could extend. You are entirely correct that I still issue a GetRequestInfo for every single request to get the Extensible fields. Unfortunately I do not know a good way to avoid this unless ICANN can add the Extensible field to the response from the /requests/all endpoint.

I just ran this again and it took exactly 2 minutes to check every request. It's longer than ideal but still much faster than what you are seeing.

I also double checked the CSV report CZDS can generate and unfortunately the extensible information is not there either.

lanrat commented 2 years ago

Just as an experiment I created a new branch extendTimeRange which stops checking zones for possible extensions if their expiration date is > 60 days in the future.

Before this change my -extend-all operation was taking ~2m, with it it takes ~18s. If I set it to just check 31 days out it takes ~8s.

I agree this is a lot faster, but I'm not yet convinced I want to merge this due to the possible problems that it could create in the future if ICANN changes the range, but you are welcome to pull from that branch and see if it helps you and report back

uggyuggy commented 2 years ago

Hi,

You are entirely correct that I still issue a GetRequestInfo for every single request to get the Extensible fields.

Ok so that is around 1200 requests. At least now we have both the same thing :) and it makes sense to me.

Unfortunately I do not know a good way to avoid this [...]

I agree, don't know either. Looks it's not possible for now, to get the status without asking each individual ID.

I just ran this again and it took exactly 2 minutes to check every request. It's longer than ideal but still much faster than what you are seeing.

Ok. Thank's. I may try to make requests from a server located in another country or another AS, just in case. I agree that is surprising to have such difference. Or does go procuded code makes more parallel requests depending number of CPUs or ?

I also double checked the CSV report CZDS can generate and unfortunately the extensible information is not there either.

Yes, agree. Unfortunately not there. My "possible improvement idea" with that "report" file, was to use last column Expire Date as a way to extract the "only" 20 TLDs (average) expiring in next 30 days. So running daily, this will send 20 requests instead of 1200 (60 times less!) for the same final result (all those that could be extended, will be extended). I'm curious if the 120sec on your side would decrease to few seconds only. Whatever would be your decision on this, I fully respect it. Just wanted to share with you, which is done.

I have not tested yet if it would work to make the "extend" request to the 20 TLDs directly, without asking first if this is Extensible: true so it would possibly save another ~ 20 "checking" requests.

Thank's

lanrat commented 2 years ago

Or does go procuded code makes more parallel requests depending number of CPUs or ?

Not by default. In this code, the only part that uses parallelization is czds-dl.

I have not tested yet if it would work to make the "extend" request to the 20 TLDs directly, without asking first if this is Extensible: true so it would possibly save another ~ 20 "checking" requests.

If there are no requests to extend then this will result in more requests than needed, which I suspect will be most of the time as you only need to request a zone to be extended once (per extension).

I also want to be as nice to ICANN's API as possible. The endpoints I'm using are not officially documented.

uggyuggy commented 2 years ago

Just as an experiment I created a new branch [...]

Great :+1: I will test it soon hopefully.

If I set it to just check 31 days out it takes ~8s.

Nice. That's pretty much what I had in mind as improvement :+1:

[...] I'm not yet convinced I want to merge this due to the possible [...]

I'm not sure to understand what kind of possible issue you have in mind, but I will fully respect.

lanrat commented 2 years ago

My "possible improvement idea" with that "report" file, was to use last column Expire Date as a way to extract the "only" 20 TLDs (average) expiring in next 30 days.

Unfortunately, in order to request an extension, you need the RequestID, which the CSV report does not include.

uggyuggy commented 2 years ago

Not by default.

Ok, thank's.

If there are no requests to extend then this will result in more requests than needed

if nothing to extend, this is 20 requests to extend, versus 20 to check (no improvement ok, but not worst) if 5 to extend, this is 20 requests to extend, versus 20 to check + 5 to extend. (so some improvement)

you only need to request a zone to be extended once

Not sure what you mean ? Many (at least for me) seems to "approve" only for like 6 months. So after 6 months it will be extended again ?

I also want to be as nice to ICANN's API as possible.

Ok I agree. Even if that would possibly improve a bit in some cases, that would not be much so, you are correct, so better to stay nice. I want to be nice with the API server too I guess, by not sending 1200 requests :)

uggyuggy commented 2 years ago

Unfortunately, in order to request an extension, you need the RequestID, which the CSV report does not include.

I'm quite confused again here, As you provide czds-request -extend string "comma separated list of zones to request extensions"

And so far it looks working great, I had no issue by requesting to extend using TLD (instead of request ID).

Sorry if I missed something here ?

lanrat commented 2 years ago

Sorry if I missed something here ?

When only a TLD is provided, and not a requestID, before we can do the requested action, I perform additional lookups to find the most recent requestID for the provided zone. Check it with -verbose to see the difference.

uggyuggy commented 2 years ago

I may try to make requests from a server located in another country or another AS, just in case.

Just tested from another VPS server on another ISP / AS (but same European country) Same time on my side: 7m36,792s :(

Will continue investigations

uggyuggy commented 2 years ago

Just as an experiment I created a new branch extendTimeRange

Thank you. Seems to be exactly what I had in mind 👍

It took 8.1s to complete ! (none candidate to be extended, so it may add few extra seconds sometimes)

...
2022/09/05 15:41:09 request "scb": "**********" expires on Wed Oct  5 23:59:59 2022, > 30 days threshold, looking no further
2022/09/05 15:41:09 requesting extensions for 0 tlds: []

real    0m8,140s

Hoping you may consider adding it to master in future ;)

Thank's!

lanrat commented 2 years ago

I decided to merge the changes from extendTimeRange into the latest release v1.2.10, with expiryDateThreshold set to 120 days. I know this is a lot further into the future than what you want, but I think it's a nice compromise. On my system it only takes 25s to run -extend-all, but leaves enough room so that if the range is changed in the future it should still work.

uggyuggy commented 2 years ago

Great news ! 👍

120 days on my side -> 48sec 30 days on my side -> 9 sec

Easy for people to change expiryDateThreshold into the code for anyone. And it's also less connections to the remote API server. All good.

Thank you very much for the open discussion.