Review by Nats - on Adoption call issues (Part 1)

[suehares]

Link for email discussion https://mailarchive.ietf.org/arch/msg/idr/202uy9M61gIN3tJdMZNLmKEQSYI/

The following review is for

• BGP-CAR-03 (draft-ietf-idr-bgp-car-03)

with respect to the intent-driven service mapping problem and how it addresses provider network deployment scenarios that are seen today, while mainly focusing on the issues raised in IDR GitHub by the CT team.

• https://github.com/ietf-wg-idr/draft-ietf-idr-bgp-car/issues

  1) F3-CAR-Issue-1: BGP-CAR Appendix A.7 – Anycast EP Scenario: 2) F3-CAR-Issue-2: BGP-CAR – Consensus on the need for resolution-schemes 3) F3-CAR-Issue-3: CAR-Q3 - Handling [of] LCM and Color-EC 4) F3-CAR-Q4: Mis-Routing in Non-agreeing color-domains for Anycast EPs 5) F3-CAR-Q5: Update Packing Observations

Having spent considerable amount of time for this review, I hope it helps operators/vendors to,

• Get closer to this problem space
• Understand how CAR achieves different aspects of service mapping
• Get implementation insights for CAR based service mapping
• How different deployment scenarios are handled by this draft

Verdict: Has Issues

[suehares]

part 2 - of discussion of https://mailarchive.ietf.org/arch/msg/idr/202uy9M61gIN3tJdMZNLmKEQSYI/

Setting up Key Aspects:

Quoting from RFC 1925: Twelve Networking Truths:

(4) Some things in life can never be fully appreciated nor understood unless experienced firsthand. Some things in networking can never be fully understood by someone who neither builds commercial networking equipment nor runs an operational network.

Before we begin, Intent-driven service mapping using BGP is a substantial effort. It applies to "all" BGP Service Families whose nexthop is reachable via transport tunnels. These tunnels belong to various forwarding architectures such as MPLS (RSVP-TE, SRTE), IPv6 (SRv6), IPinIP or MPLSoverUDP. It introduces new constructs that allow for an operator to be able to specify the service intent. Therefore, this is a substantial upgrade for existing provider network deployment scenarios such as MPLS-VPN Options, CsC, Multi-Homed CE and Anycast scenarios that are prevalent today. Therefore, this review will evaluate BGP-CAR based on certain key aspects and operational criteria.

Intent-Driven Service Mapping Key Aspects:

To better understand the nature of issues raised, it is important to consider the basic intent-driven service mapping aspects.

A) Classification & Grouping: Construct to group underlay tunnels with sufficiently similar TE characteristics

• CAR: Classification: Color Grouping: None (Not clear from the draft if any construct is used)

DR# Color is the construct for an intent, already well-defined as per RFC 9256 - described in the introduction and main sections.

Hares RFC9256 references the Color-EC is defined in RFC9012. The color usage is defined in RFC9256. Color is well-defined for SR (Spring) usage in RFC9256. The definitions in draft-ietf-idr-bgp-car specify expanding this to the car routing solution. Stating it is "well-defined" needs to be clarified to "well-defined for the SR routing solution".

B) Resolution & Steering: Construct for overlay routes with a Mapping Community e.g. Color-EC, to resolve next-hop reachability to the underlay path with the same "effective color" arrived from the following

• CAR: Resolution for CAR routes and SR policy based steering/automatic-steering for Service routes
• Color-EC on Service Route (Mapping Community)

• CAR Type-1: IP: Color as key in NRLI

  • CAR Type-2: Colored IP Prefix via a) Color-EC on CAR route for Anycast, and b)LCM-EC for non-agreeing color-domains

  and download service routes to FIB when its path is reachable

Hares Sections 2.5, 2.10, and Appendix B deal with the different types of color. As noted in the shepherd's report, enhancements for Sections 2.5 and Section 2.10 were suggested for -06. Further specific suggestions for improvement for these sections can be made.

C) BGP Extensions: (AFI/SAFI Encoding/Decoding) A BGP family to extend the above constructs to adjacent domains (AS/IGP)

• CAR NLRIs: (New NLRI formats for CAR and VPN-CAR)

  Type-1: Color as key in NRLI Type-2: IP Prefix as Color Color-EC attr (for Anycast) LCM-EC attr (for non-agreeing color domains) IP prefix for CAR types and RD:IP prefix for VPN-CAR types

Hares Sections 2, 7, 8, 9, and Appendix B deal with the different types of color. As noted in the shepherd's report, enhancements for sections 2, 7, 8, and 9 were suggested for -06. Further specific suggestions for improvement for these sections can be made based on -06. This simply seems to be a list of changes defined by the solution.

D) Path Availability & Selection: This is the final and most important aspect which has two important steps for selecting transport routes for nexthop reachability:

i) Avoid path hiding - CAR route: Type-1: Unique "IP: Color" key in NLRI Type-2: Unique "Color IP Prefix" per color-locator

ii) Path selection and Resolution should use the same "effective color" CAR route Type-1:
Path Selection key Prefix: IP and Type-1-NLRI-Color Resolution key Prefix: IP and "Effective Color" (Type-1-NLRI-Color or Color-EC on CAR route or LCM-EC on CAR route)

CAR route Type-2:
     Path Selection key Prefix: IP color prefix
     Resolution key Prefix: Not needed as it is routed

[suehares]

Part 4 of discussion of https://mailarchive.ietf.org/arch/msg/idr/202uy9M61gIN3tJdMZNLmKEQSYI/

F3-CAR-Q1: Status = Unresolved

BGP-CAR Appendix A.7 – Anycast EP Scenario: https://mailarchive.ietf.org/arch/msg/idr/nAj25sX0x_lp09VEqUDSCxmDR_w/

Note: This is not a solution but just a workaround. While bug fixes in code are common, it is painful to see bug fixes and work-arounds in new and evolutionary technology drafts.

F3-CAR-Issue-1.1: To work around Issue-1.2 and Issue-1.3, each anycast end-point is configured with a different CAR-NLRI-Type-1 'Color' to avoid path hiding and attaching the same Color-EC to all EPs additionally.

• Does this sound familiar? Of course, Prefix remains the same. CAR-NLRI-Type-1 'Color' now becomes the RD and Color-EC becomes the intent. In CAR, this comes with a lot of local policy overhead @ PE/ASBRs to STOP automatic resolution using CAR-NLRI-Type 'color' as intent and replace it with Color-EC as "intent".
• This is the reason for the first two quotes mentioned in procedure (III.) and quotes in procedure (I.). This issue still exists, and the draft needs to call out policy overhead required in PE/ASBRs.

DR: Color in the NLRI represents the route intent. The use of Color-EC provides an automated mechanism to resolve the next-hop and takes precedence over route color as the text you’re quoting clearly states.

Hares: The -06 text should be examined to determine if change provide the clarity in the procedure description on: a) next-hop resolution with Color-EC, and b) precedence in resolution (Color-EC, LCM-EC, and color in NLRI (if type-1)).

F3-CAR-Issue-1.2: Considering the second quote in procedure III. and an Administrative domain with ColorSRTE based transport tunnels, a CAR route can resolve over a Color SRTE route with TEA. This means that for N anycast EPs, there are N colors assigned and hence N color SRTE paths per AS domain for each Color-EC. This multiplies the Color SRTE forwarding state by a factor of N. This needs to be called out in the draft.

DR: All the N anycast routes can resolve via the same SR-TE color, specified in the color-EC. This illustrates the benefits of the decoupling and indirection between route and next-hop colors.

Hares: Please cite the section that describes the procedures for resolution in -06.

F3-CAR-Issue-1.3:
Quoting from section A.7 "Both E2 (in egress domain 2) and E3 (in egress domain 3) advertise Anycast (shared) IP (IP1, C1) with same label L1"

• Labels are dynamic. If the same label has to be sent on both, there needs [to be] some static configuration to coordinate this. This should translate to a SHOULD clause in the procedures. If an implementation is unable to do static labels, then recommendations need to be provided to address the same.

DR: Not sure what is specific to CAR SAFI here. Static MPLS labels / SR prefix-SIDs / anycast SIDs are all existing well-known features.

Hares: Do you state a requirement for static labels in -06 of the draft? I find static labels discussed in section 5.5.2. I find the word label is sections 2.9.2.1 (Label TLV), 2.9.2.2 (Label index), section 5.3, section 8, section A.3.2, C.1 and C.2 (no labels). Would you let me know which section covers this issue? Hares:: Issue needs to be resolved prior to WG LC.

F3-CAR-Issue-1.4: How does ECMP and protection work here?

• Based on procedure (I.), a local policy can be used to discard CAR-NLRI-Type-1 'Color' and use alternate colors, say in Color-EC to resolve the CAR route.
• However, the question that still remains is how the route needs to be maintained in the RIB? Should the route be installed as IP:C1, IP:C2? Since these prefixes are unique, path selection cannot act on them. This is another example of C1 and C2 used as "distinguisher" rather than "intent" and hence the ambiguity due to overloading.
• If they are installed as just IP, which RIB they are installed to? This needs to be explained in the draft.

DR:Section 2.7 describes the key is (E,C), so multipathing is automatic, does not require additional “import” as in CT.

Hares: This is described for BGP CAR SAFI Type 1 Route (E ,C) in section 2.7 in version -06. The multipathing in section 2.7 refers to advertisement within a single AS between two Area Border Routers (ABR). Is the intent of this section a single AS? Or do you need to apply this to multiple AS in a path between egress AS and ingress AS.

Hares: Should be discussed prior to WG LC

DR: Appendix A.7 provides examples of two choices. Not sure what is ambiguous. [The] choice of color and control for specific intent use-cases is in the hands of the operator. Hares: Question for Nats: Is section A.7 clear? If not, what is unclear. If A.7 is clear, do you have a procedure in the main text that explains A.7?

F3-CAR-Issue-1.5: Section B.2 - N:M distribution

In Figure 12: N:M illustration, attaches at least two Color-ECs on the CAR route while overloading the CAR-NLRI-Type-1 'color' as both distinguisher as well as intent for unicast EPs. There are no procedures as to which Color-EC should take precedence in which AS/ABR domain. This needs to be clarified. It should also note that there needs to be one additional Color-EC on the CAR route for each M (M1, M2, ...).

Hares: I want to verify that this example follows the description from 1/29/2024 presentation: a) Color-EC is used to resolve NextHop and b) NLRI Color (E,C) is used to indicate the intent of the traffic. There is no LCM-EC in this example. Hares: - Example in B.2 needs to be discussed.

DR:The example clearly illustrates how the entire e2e resolution can be automated by leveraging the power of hierarchy and the standard and widely implemented/deployed Color-EC mechanism from RFC 9256.

• In this example, the color assignments for domains across the network are intentionally and by design disjoint, hence the resolution will always occur deterministically regardless of the order.

Hares: A discussion of how your comments on this example match your 1/29/2024 presentation would help the shepherd.

F3-CAR-Issue-1.6: Section B.2 - N:M distribution and Anycast In Figure 12: N:M illustration, now with Anycast in play and as per F3-CAR-Issue-1.1, there will be atleast "three" Color-ECs, where Color-EC-Anycast represents Anycast plus the "two" Color-ECs for the N:M distribution. As per A.7, CAR-NLRI-Type-1 'color' now becomes the distinguisher and Color-EC-Anycast becomes the intent. This is a "critical" issue because there is no clear precedence in picking the Color-EC assigned for Anycast. This needs to be clarified as part of the draft.

Hares: Version -06 does add text to clarify the precedence between colors (LCM-EC, Color-EC, and (E,C)). However, -06 version only has 2 colors (Color-EC and (E,C)) in diagram B.2. A review of the

DR: The Color-EC assignments need not change if the route is for an Anycast or Egress Domain Prefix. So there is no issue above.

Hares: Are you stating that in the example in Appendix B.2, that the routes (E,C) remain the same? This means (E2,100), (E2,200), (E2,300), and (E2,400) remain the same across the domain? The next hops are changing for this route (E2-> 231 -> 121)? Does -06 additional text with precedence help this example?

F3-CAR-Q2: Status = Unresolved BGP-CAR – Consensus on the need for resolution-schemes https://mailarchive.ietf.org/arch/msg/idr/g6ZCJYzWwgRsilWlZY74MTv9Fqk/

F3-CAR-Issue-2.1: Based on the above issue list, especially Issue-1.1, Issue-1.4, F3-CAR-Issue-1.1, 1.4, 1.5 and 1.6, it is indeed important for CAR to have a clear scheme for resolution that handles all the of these issues.

• The nature of this scheme needs to be explained as part of this draft specifically, that vendors can use to incorporate during implementation. Offloading this to a local policy blob is prone to make implementations incompatible due to configuration ambiguity.

DR: None of the above comments raised any specific issue that were not discussed previously or has not been addressed in the draft. Hope my responses have provided clarity. If some point is not clear, please let me know.

Hares: Which sections deal with route resolution in the draft? This sections can be reviewed specifically in version -06.

F3-CAR-Q3: Status = Unresolved CAR-Q3 - Handling [of] LCM and Color Extended Communities https://mailarchive.ietf.org/arch/msg/idr/w5ROKVQPtVcI_BTBXfnKpKB4h4k/

F3-CAR-Issue-3.1: Considering quotes from section 2.10 in Procedure (III.), F3-CAR-Issue-1.1 to 1.6 apply. Additionally, LCM-EC needs to be factored into F3-CAR-Issue-2.1.

F3-CAR-Issue-3.2: Considering the following quotes from sections 2.5, 2.9.4, and 2.10 in Procedure (III.) respectively,

• "For a CAR route, Color-EC color takes precedence over route NLRI color."
• "When a CAR route crosses the originator's color domain boundary, LCM-EC is added."
• "If present, LCM-EC is the effective intent of a BGP CAR route."
• "Default order of processing for resolution in presence of LCM-EC is local policy, then BGP Color-EC color, and finally LCM-EC color"

These normative text contradict each other. If Color-EC takes precedence over CAR-NLRI-Type-1 'color', then Color-EC becomes the "intent" and CAR-NLRI-Type-1 'color' becomes purely the distinguisher and not overloaded with intent. In such cases, it is not very clear whether Color-EC value should be the effective intent instead of NLRI color and which one should be copied into LCM-EC since, CAR-NLRI-Type-1 'color' is just a distinguisher.

This needs to be clarified as part of the draft text.

DR: The text clearly states the above order of precedence is for CAR route resolution. Color-EC when present represents the intent/color used for resolving the BGP CAR next-hop, following semantics already established in RFC 9256. If not present, resolution uses the route color (whether its’ in NLRI or LCM). Let’s try to avoid cherry picking text out of context and mixing them.

Hares: Section 2.5 in version 06 states: "Local policy takes precedence over color-based automated resolution. For a CAR route, Color-EC color takes precedence over route NLRI color. When LCM-EC is present, Color-EC color takes precedence over LCM-EC color." Does this clear up the precedences?

Hares: Query needs to be made to Nats.

F3-CAR-Q4: Status = Unresolved BGP-CAR – Mis-Routing in Non-agreeing color-domains for Anycast EPs https://mailarchive.ietf.org/arch/msg/idr/OOZOBSyjdAYBar8NxvOqo6-5fAc/

F3-CAR-Issue-4.1: Quoting from section 2.9.4 (version -03) 

 "If two BGP paths for a route have different LCM values,
  it is considered an error and the route is not
 considered for bestpath selection."

The above text still does not address the misrouting problems that might arise in Anycast with Non-Agreeing color domains as indicated in the above issue link.

However, this seems to have been "worked around" using Color-EC as the intent, CAR-NLRI-Type-1 'color' now acting solely as a distinguisher and hence its derived LCM-CE also acts as a distinguisher in the receiving non-agreeing domain. The local policy operations for this scenario needs to be clearly specified to avoid ambiguity in the draft. It is also important to note that a portion of colors from "intent namespace" is being used purely as distinguishers and others being overloaded with intent as well. This needs to be clarified in the draft as to how the intent namespace(s) are managed so that local policy schemes can derive how this namespace is used.

DR: I’ve already addressed this above and previously.

• NLRI color identifies the route intent, which includes the intent for Anycast or Egress visibility.
• Text in manageability/operational section provides relevant context for an operator. Everything else is as per already described procedures and discussions captured previously. Hares: Section 11 on "Manageability and Operational Considerations" points to the fact the LCM-EC will be modified at the edge of Color domains. Your description at the Interim on 1/29/2024 on the LCM color changing value at each different color domain was helpful. In the following paragraph:

" When color-aware routes propagate across a color domain boundary, there is typically no need for coordinating color assignments, since the IP prefix is unique in the transport network, and hence makes the color scope also unique and non-conflicting. The color only needs to be re-mapped into a local color assigned for the same intent (which is carried in the LCM-EC)."

It would be useful to point to an example with LCM-EC changes for Type-1 and Type-2. Also, it would be good to have the example of Color-EC (Appendix B.2) mentioned as an example of Color-EC resolution.

F3-CAR-Q5: Status [is] Resolved Update Packing Observations are considered resolved. No additional text is needed.

DR: Thank you.

[suehares]

Part 3 of discussion of https://mailarchive.ietf.org/arch/msg/idr/202uy9M61gIN3tJdMZNLmKEQSYI/

Review [By Nats] (follows comment 2)

Switching back to the open issues and scenarios, this review will focus on evaluating CAR on the above set of aspects and criteria.

MAJOR: "Where is my color?"

• From CAR-00, as a draft that addresses this "new" problem space of using "Color" to achieve "Intent driven service mapping", CAR introduces the "Where is my color?" ambiguity in procedures for any node receiving and processing a CAR route.

DR: The draft clearly specifies where color is derived from and for what purposes. Specific comments are addressed below.

Hares: As pointed out by my Shepherd's report on 1/26/2024, the text regarding the usage of LCM-EC and Color-EC needed upgrading. A discussion of this point after -06 to see if the improvements have addressed the issue is warranted.

These are arrived from the following procedures:

PROCEDURE I: CAR Route validation

Quoting from section 2.4: BGP CAR Route Validation _[version-03]_ 
  "A BGP CAR path (E, C) from N with encapsulation T is valid if color-
   aware path (N, C) exists with encapsulation T available in dataplane."

  " A local policy may customize the validation process:
   *  the color constraint in the first check may be relaxed: instead N
      is reachable via alternate color(s) or in the default routing
      table"
      ...
  "A path that is not valid MUST NOT be considered for BGP best path
   selection."

Hares: Feedback is needed from Nats on specific improvements to -06 text for 2.4.

PROCEDURE II: Path Selection and Availability

Quoting from section 2.7: Path Availability

  "The (E, C) route inherently provides availability of redundant paths
   at every hop, identical to BGP-LU or BGP IP."

Hares: Feedback is needed from NATS on what is unclear.

PROCEDURE III: CAR Route resolution

Quoting from section 2.5: BGP CAR Route Resolution

  "For a CAR route, Color-EC color takes precedence over route NLRI color."

  "A BGP CAR route may recursively resolve over a BGP route carrying
   Tunnel Encapsulation attribute. Procedures of section 8 of [RFC9012]
   apply in presence of BGP Color EC in the CAR route. They are
   extended to use LCM EC and Color in CAR route NLRI as per above and
   Section 2.9.4 in absence of BGP Color EC."

Quoting from section 2.9.4: LCM-EC "When a CAR route crosses the originator's color domain boundary, LCM- EC is added."

  "An implementation SHOULD NOT send more than one instance of the LCM-
   EC.  However, if more than one instance is received, an
   implementation MUST disregard all instances other than the one with
   the numerically highest value"

  "If present, LCM-EC is the effective intent of a BGP CAR route."

  "LCM-EC Color is used instead of the Color in CAR route NLRI for
   procedures described in earlier sections such as route validation,
   resolution, AIGP calculation and steering."

Quoting from section 2.10: LCM and BGP Color Extended Community usage "Default order of processing for resolution in presence of LCM-EC is local policy, then BGP Color-EC color, and finally LCM-EC color"

This CAR route will have one or more of the following depending on the NLRI type and the Scenario being addressed.

Item | CAR color variable | Scenario "=====================================" | A. | Color-EC color | Anycast, Single Domain N:M color | B. | LCM-EC color | Multiple color domains | C. | CAR-NLRI-Type-1 'color' | Default, used for path availability and selection additionally | D. | CAR-NLRI-Type-2 'IP Prefix' | Defined in Section 2.9.3, 8, 9 and 10.2 "=====================================================" item | CAR color variable | Scenario** "=====================================================" | A. | Color-EC color | Anycast, Single Domain N:M color
| B. | LCM-EC color | Multiple color domains
| C. | CAR-NLRI-Type-1 'color' | Default, used for path availability and selection additionally | D. | CAR-NLRI-Type-2 'IP Prefix' | Defined in Section 2.9.3, 8, 9 and 10.2 "====================================================="

Hares As noted in the shepherd's review, clarifications on the processing and handling of LCM-EC and Color-EC are necessary for the -05 text. A walk-through of the -06 should be made to determine if any further work should be done.

NATs: Quoting from RFC 1925: Twelve Networking Truths:

(3) With sufficient thrust, pigs fly just fine. However, this is not necessarily a good idea. It is hard to be sure where they are going to land, and it could be dangerous sitting under them as they fly overhead.

Issue-1.1: As mentioned in key aspects, the path selection key and the resolution key can be different depending on the scenario. This will cause misrouting issues such as the ones mentioned in F3-CAR-Q4 below. [Some] specific text needs to be added to address [this] scenario.

DR: There is no practical misrouting issue as has been extensively discussed during the WG adoption call. See the summary of a substantive discussion between Bruno and Jeff. https://mailarchive.ietf.org/arch/msg/idr/ZKcFdSSn02hFTqe6p4pNOgNjDYY/

DR: As discussed there and recommended by Jeff, we’ve added relevant text in the Operational / Manageability considerations [in] section 11.

Issue-1.2: CAR-NLRI-Type-1 'color' is overloaded for path selection, path availability as well as intent. While this solves path hiding for Unicast EPs, there is path hiding @ ABR/ASBRs in anycast and multihomed EP scenarios, and as a result, all paths are not visible to the ingress domain and thus end-to-end, as both Prefix and Color are [the] same in all paths. Therefore, overloading CAR-NLRI-Type-1 'color' as both the distinguisher (RD) as well as "Intent" needs to be explicitly called out for each scenario.

DR reply

• “Anycast” requires the network to route to the closest endpoint, and this is naturally and inherently supported with an (E,C) model. Reference Appendix A.7 [in] Example 1.
• For a specific intent requirement that the paths to specific egress domain/nodes be visible to an ingress PE, assigning a specific color for the egress domain/node is aligned with the intent, not overloading. This would be Appendix A.7 example 2.
• This was also discussed and noted back during Jeff’s initial comparison. Moreover, using EBGP add-path at ASBR to get around this will use AddPath-ID as the distinguisher which is again, not well deployed and limits the distinguisher to the scope of the EBGP session and not end-to-end (e.g. RD)
• CAR doesn’t mandate any use of Addpath-ID more than what CT does. Let’s please stop attributing these gratuitous claims. https://mailarchive.ietf.org/arch/msg/idr/gQ2QX8RNzu7jQTJ0mawobNtledI/

Issue 1.3: Migration from unicast to anycast is not seamless for E2E path availability use-cases.

• This can happen when a CE is multihomed to another PE in the egress domain. Operators should expect path hiding and will have to use A.7 workaround to have an additional variable, Color-EC which is attached to the CAR route, and is preferred over CAR-NLRI-Type-1 'color' as the effective intent even in single color domains.
• This is further discussed in F3-CAR-Q1 below.

DR reply

• To clarify, the Color-EC indicates the intent/color to be used for next-hop resolution, following the well-established semantics in RFC9256.
• It provides flexibility to automate the resolution when the color for resolving the next-hop is different than that of the route, as in this case.

Issue 1.4: Colorless Transport Tunnels with varied intents like RSVP-TE, MPLSoUDP, and IPinIP are quite prevalent in existing brownfield [networks] and customers do foresee these deployments continuing into the next decade as well.

• However, there still needs to be [constructed] to achieve the aspect of classification and grouping in such deployments. How can local policy achieve this in the absence of 'transport color' and 'TEA'?

DR: Color is indeed the construct for resolution, augmented by local policy to map as needed to traditional, color-unaware mechanisms. Please see Section 2.5.

[suehares]

Shepherd has yet to review this for editorial issues.
Technical issues are considered closed.