Closed bluikko closed 2 years ago
Thanks a lot @bluikko, I’ll go over your suggestions one at a time and see what I can do to make the IX-F data handling better.
Hi @bluikko, thanks again for the suggestions.
I've just pushed a new commit to the dev branch in which I've introduced a new option for the ixf-member-export
command: --merge-file
.
You can read more about it in the changelog from dev and in the staging instance of the docs web site.
I think this new feature could help with the top 2 bullet points from your message above: to add more info about the IXP and to add peering VLAN prefix information. (The URL has been fixed in another commit, unrelated to this new feature.)
The CI/CD pipeline is now running, and if everything goes well (🤞) a pre-release should be out soon on the test instance of PyPi: v1.12.0-alpha1
It'd be great if you could give it a try, by following the instructions reported on the docs about how to install pre-releases: https://arouteserver.readthedocs.io/en/latest/INSTALLATION.html#development-and-pre-release-versions
Regarding the 4th bullet point, as you said it's not trivial. I'll take a better look at it in the next days, however at the moment I can't see a way to solve it with the information present in ARouteServer.
Thanks
I've just pushed a new commit to the dev branch in which I've introduced a new option for the
ixf-member-export
command:--merge-file
.
So arouteserver does a dict merge between the generated result and the merge file. That sounds great.
Will test it soon.
For the 4th point, I've found out that in PeeringDB each of the IP addresses indeed shows as a separate interface. But it is possible to manually edit that data at PeeringDB after it has imported the IX-F data. So seems the feedback was given in order to avoid this manual editing.
It works excellently. It took me a few runs to realize that the ixp_id
attribute in the merge file cannot be zero (it would be 0 by default if not otherwise defined by the arg). Otherwise an error (+ error handling the exception) is encountered.
The same non-zero ixp_id
must be passed on the command line: otherwise arouteserver will use ixp_id
0 and not the ixp_id
in the merge file.
This was not very logical and could probably be either documented or improved. The first example JSON file in documentation also uses ixp_id
of 0.
I would propose to update the documentation
"support_contact_hours": "24x7",
to
"support_contact_hours": "24/7",
Since the JSON schema example gives "24/7" and not "24x7".
And a second nit, on the switches:
usage: arouteserver ixf-member-export [-h] [--cfg FILE] [--cache-dir DIR] [--logging-config-file LOGGING_CONFIG_FILE] [--logging-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}]
[--clients FILE] [--ixp_id IXP_ID] [--vlan-id VLAN_ID] [--merge-file MERGE_FILE] [-o OUTPUT_FILE]
shortname IXF_ID
The ixp_id
uses underscore while the arg vlan-id
(+ others) uses a dash. I don't know if it breaks too many people's systems but for consistency maybe it should be ixp-id
(since the other such args use a dash). The argument could also be said for making vlan-id
instead use an underscore to keep them consistent with the JSON schema instead of other args. Maybe it needs to wait until a major release that can have breaking changes.
Edit: when the merge YAML file has phone numbers, for example:
emergency_phone: +1235556789
The value in the result JSON file is incorrectly type int and not a string. Of course this can be fixed by changing it to
emergency_phone: "+1235556789"
but it is a bit counterintuitive since YAML does not usually need quotation marks. Don't know if this behavior should have change to force phone numbers to strings.
Also, the IX-F JSON examples list phone numbers with spaces so perhaps the documentation merge file example could also group the phone digits with spaces, e.g. "+123 456 789"
.
Similarly to the "prefixes" in the documentation merge file example, the IX-F JSON examples list prefixes in format "192.0.2.0"
and "2001:db8::"
without the masks (the masks are only in the mask_length
attributes.
And yet another small suggestion: arouteserver prints a line every time about checking for updates. This could be a good opportunity to print the current version.
ARouteServer 2021-11-19 01:52:28,693 INFO Checking latest version
Thanks for testing it @bluikko, and for providing all these comments.
I've added several additional sanity checks against the well known keys for the ixp_list object; they should help the operator to figure out issues and inconsistencies between the values used by ARouteServer to build the member list (which are those from the CLI) and any eventual conflicting value from the merge file. I've also fixed the example used for the doc, which was wrong of course due to what you said.
support_contact_hours
example has been fixed, and also the phone number format. However, I'm not planning to add code to handle the YAML-to-JSON translation of it, since it's really a corner case due to YAML, and the user can easily fix it by quoting the value.
IP prefixes have also been fixed in the example, removing the mask.
I've also implemented a deprecation for --ixp_id
: I'll use the approach of having all "dash"-based options, which is more oriented towards the GNU way of handling CLI arguments.
The pipeline for a new alpha2 release is running; it'd be great if you could give it a try (assuming it will succeed!)
Thanks again.
Also, the tool now shows the current version while checking for any new one.
The pipeline for a new alpha2 release is running; it'd be great if you could give it a try (assuming it will succeed!)
Tested with alpha5 and it seems to work well. Thanks for your work for creating a viable option for IXP Manager, it is very important for avoiding a total monoculture.
Hello @bluikko, I've just released 1.12.0, which contains the improvements/bugfixes for most of the points from this ticket.
I've also created #91 and I've tagged you there: in this new ticket I've captured your comment about the individual VLANs in connection_list
. My idea is to attack it as part of a different ticket (assuming I'll find a way to work it out 😄) I hope it makes sense for you.
I'm going to close this ticket now, feel free to reopen it if you find something wrong in the new release or if there's anything missing. Thanks for your feedback!
We recently enabled the IX-F member export JSON schema generated by arouteserver and PeeringDB had some suggestions to make the data more useful. Some of the suggestion below come from that feedback. We could do these by post-processing the generated JSON file but I would expect anyone using this arouteserver feature could benefit and PeeringDB could need to send less mails if the features were provided by arouteserver out of the box. It could also promote the usage and usefulness of the JSON schema.
ixp_list
attribute: a file could be included in this section so that IXP could define itscountry
, fullname
,peeringdb_id
,url
, various[...]_email
attributes, etc. Or even a list ofswitch
es. Instead of adding a bunch of options to arouteserver to include various extra data, the obvious way would be to just include a named file so that operators can include whatever data they want in theixp_list
. Perhaps it could be a switch passed to arouteserver that enables inclusion of a named file passed to the option, and rendered after the currently generatedixp_list.vlan
, just before the end ofixp_list
.ixp_list.vlan.ipv4
,ixp_list.vlan.ipv6
) to thevlan
list (ixp_list.vlan
) would be much more difficult since arouteserver does not really have this information; the closest is theLOCAL_PREFIXES
information. That option could have much more than just the peering VLAN prefixes; maybe it could be cross-checked with the IP addresses assigned to members and then pick those prefixes fromLOCAL_PREFIXES
that have IP addresses assigned to members/peers.vlan
s inconnection_list
: according to PeeringDB the current way to list each member IP address in its ownvlan_list
element could be understood as each IPv4/IPv6 address being their own interface (=IXP port). The trouble here I think is that arouteserver does not really have the interface/port-level information and could not associate each IPv4 address with the corresponding IPv6 address. Currently we just list IPv4 and IPv6 addresses inclients.yml
without any way to distinguish which port they belong to. Individual addresses within the same address family (multiple IPv4 addresses/multiple IPv6 addresses) muddy the waters here - should they be kept in their individualvlan_list
elements? It seems to me so and that would make the difficulty. How to associate IPv4 address with its IPv6 counterpart to put them in the samevlan_list
element, while keeping a separatevlan_list
element for some other IPv4+IPv6 pair? It could be much easier if all IPv4+IPv6 addresses could be lumped in the samevlan_list
element but I am not sure if this would still be any better to PeeringDB than having all of the IP addresses in their own individualvlan_list
elements. Will update if I get a response to that question. Example:{ "_comment": "New definition", "asnum": 65520, "name": "Member", "connection_list": [ { "ixp_id": 0, "vlan_list": [ { "vlan_id": 0, "ipv4": { "address": "192.0.2.1" }, "ipv6": { "address": "2001:db8::1" } } ] } ] }