Add `relatedConsignments` to consignments

[bmeesters]

Type of request

• [x] I'd like to add something to the OTM 5 spec
• [ ] I'd like to change something in the OTM 5 spec

Is your feature request related to a problem? If so, please describe the problem. For example: with the current specification I cannot model [...]. Or: I'm unable to provide the use case [...].

There are a number of advanced use cases for working with consignments (see also this discussion).

• A requested consignment is oversized and cannot be transported as one. Thus the solution is to split the original consignment into multiple smaller consignments which will be transported by different vehicles at different times.
• Something went wrong and part of the goods could not be delivered (for example they are damaged). There needs to be a follow up consignment with new goods to replace the old goods.

In each of these cases there are multiple consignments that are related together. Currently OTM5 does not properly support linking multiple consignments together somehow.

Describe the solution you'd like What would you like to add or change to the specification? Please be as clear and concise as possible.

The simplest solution would be to add a new field relatedConsignments to consignment so you can refer to other consignments (that have a different status).

Describe alternatives you've considered Did you try to solve it with the current specification? If so, how? Please be as clear and concise as possible.

The only other possible workaround is to use the external attributes. Otherwise another field could be added that achieves somewhat the same.

Additional context Add any other context or screenshots about the feature request here.

The biggest problem with the proposed solution is that it does not provide a good way to communicate how the relatedConsignments are related. So we might need to add another field on Association, or use a different type of association to add this.

[thomaskolmans]

At Transportial we're for this change!

[bmeesters]

@thomaskolmans, great, thanks for letting us know! Any ideas on how to describe the relationship between the consignments. E.g. add a new field on association like role for actors? Or maybe add a new field relation for all types of associations and replace role?

[BobZuidhoek]

Hi Bas,

Just some additional context on how we see this happening in our companies nowadays.

We sell boxed products (flowers/plants) to large retailers. They are packed onto pallets/trolleys. Goods are shipped from our warehouses. We do not perform road transportation ourselves, this is outsourced to road freight companies that have perishables transport (refrigerated transport) as core competency. Orders can be a single pallet/trolley, but orders of 50+ pallets or trolleys are also common.

The process currently works as follows...

We receive purchase orders from our retailer clients (which are distribution center level orders). We book these orders in as sales orders in our ERP-system. This may look something like:

• Product A 500 boxes.
• Product B 300 boxes
• Product C 200 boxes. Total 1000 boxes. We also know the pallet loadfactor is 25 boxes per Euro-pallet. So the total volume in pallets is 1000/25 = 40 Euro-pallets. We also know the maximum regular trailer capacity is 33 Europallets.

Therefore we create 2 consignments in our ERP-system. Consignment 1:

• Product A 500 boxes.
• Product B 300 boxes
• Product C 25 boxes. This totals 825 boxes (33 pallets).
  Consignment 2:
• Product C 175 boxes. This totals 175 boxes (7 pallets).
  Note that we do not split the sales order, but we now have 1 sales order and 2 related planned consignments.

The consignments 1 and 2 are sent to our road freight carrier as bookings (individual transport orders). If the carrier accepts the bookings, we're good to go. But it may happen that a request for splitting comes back from the carrier's planning department, something like "please split consignment 1 into 20 and 13 pallets". This is where things get a little dirty:

• We now rework consignment 1 in our ERP-system into 2 consignments (let's call these new consignments 3 and 4).
• We drop the transport order of consignment 1.
• We send new transport orders for consignment 3 and 4.

What you see happening here conceptually, is that an interactive booking process is taking place. From what I witness at our companies, this is a very common process to happen between shipper and carrier. The booking process has an impact on the resulting transport orders.

Note: in our situation, carriers are not allowed to split consignments during transport. Therefore, the booking/splitting process is always an upfront process (and consignments are planned and packed as such). We ourselves are knowledgeable that 1 sales order is being executed as 3 consignments (therefore we know the "relatedconsignments" internally). I can imagine different processes exist at other companies where the carrier splits consignments during transport (this scenario seems a complicated one I think).

I think it makes sense to first have a good look at how these booking processes (and the corresponding back-and-forth-processes) should work in OTM, before jumping into adding a concept such as relatedconsignment.

[bmeesters]

Thanks for the extensive details Bob. It definitely helps to look at the different processes to see how we want to change OTM. Though from what I can see the relatedConsignments would also help in your process right? I think the main addition is that when a consignment 'evolves' over time into multiple consignments you would want to relate these together right? I cannot really imagine that some consignment is being split into multiple consignments and the party issuing the order cannot relate what used to be to the new situation. But I might be missing something?

If you have any alternative ideas I would gladly hear them.

[BobZuidhoek]

Hi Bas,

It probably doesn't matter really, but from the perspective of the carrier the consignments are seen as unrelated. The carrier essentially does not know that we have 1 sales order overspanning the consignments that he knows of. Only we as shipper need to know how things are related. Therefore, we essentially don't tell the carrier of consignments being related.

I could imagine that you need the relatedconsignments functionality to for example profit from bracketed price agreements (Dutch: staffels). In the case of the example I described, if you would have a beneficial full truckload tariff, you need to add up the volume of the three related consignments to calculate the right price bracket. Within our companies, we have removed price bracketing completely, because it usually gets in the way of efficient ordering and invoicing processes. We now just have flat prices per loading metre, and agree prices based on historical transportation volumes. This simplifies processes a lot.

By the way, do you think our way of working is also as intended for OTM? By that I mean that the booking process is a simple create/delete process of transport orders until things settle?

[bmeesters]

It probably doesn't matter really, but from the perspective of the carrier the consignments are seen as unrelated. The carrier essentially does not know that we have 1 sales order overspanning the consignments that he knows of.

I can imagine something like this being more common.

However, I think the current proposal is good enough because:

1. The new relatedConsignments field would be optional. So if a party doesn't know it is not required to fill it. Also for some consignments it would not make sense, since there might not be any related consignments to begin with.
2. If a party knows the related consignments but does not want to share them because the receiving party doesn't need to know it is also not required to send them. Tackling the case above.
3. It is an easy change, just one optional field consisting of data types already available in OTM5 (associations and consignments).
4. There is currently no alternative to model consignments changing over time and relating them to their previous versions.

If there are any oversights I might be missing I gladly hear them. We will also check with some other parties if they have an opinion on this.

By the way, do you think our way of working is also as intended for OTM? By that I mean that the booking process is a simple create/delete process of transport orders until things settle?

Yes, I think your modelling makes sense. If consignment 1 cannot be transported by the carrier there are only two options that I can see:

1. Mutate the consignment 1 and create a new consignment 3 (assuming 2 already exists)
2. Cancel consignment 1 and create new consignments 3 & 4 (your current model).

IMO the second one is cleaner since you can more easily keep an audit trail of what happened. if we add the relatedConsignments it would also become easier to see what ultimately happened with consignment 1 (internally, or with parties that care). But again, you would not be required to if it is not necessary.

[BobZuidhoek]

Hi Bas,

OK thanks. Good to see we're on the right track then.

I was pondering about this over the weekend and this question came to mind: if now the relatedConsignments concept is going to get its place within Consignments, what is then still the functional use of the TransportOrder layer in the top op the tree? It seems the TransportOrder is losing its value, for example:

1. • A TransportOrder groups together 1 or more Consignments. But that's probably done better using relatedConsignments. By using relatedConsignments, this also ensures a Consignment is understood as a fully independent object and essentially has nothing to do with the overarching TransportOrder. The Consignment is probably already meant to be a fully independent object, but when you think in actual messages, I can foresee a lot of misinterpretation of happening here around TransportOrder (e.g. there is 1 Transportorder with 3 Consigments and the shipper wants to update only consignment 2, then he wrongly issues a whole update of the Transportorder with all 3 consignments, leading to confusion at the Carrier company) .
2. • The actors can be set top-level on TransportOrder. But they can also be set sub-level on Consignment. What is the benefit of doing this in the top-level TransportOrder? Is it just efficiency, so you don't have to mention them in Consignment? What happens if you would specify different actors in 1 of the Consignments than as specified in the TransportOrder (would that even be allowed)? Would it not be simpler if you only allow actors on Consignment?

3. For Constraint, the question is essentially the same as question 2.

In your opinion, is there still any business value left of having the TransportOrder overarch Consignments? My opinion would be that if there is no logical place for TransportOrder anymore, that it should be scrapped. It would only create an expensive layer that needs to be built and dealt with in many systems.

Maybe others can also chime in on business scenario's they encounter how this is being used.

[bmeesters]

This is a good question. In principle the TransportOrder makes it easy to send a bulk request, where a party can send all consignments that need to be transported (and might be billed together for efficiency). In that sense it is different from the relatedConsignments in that the latter is only meant to link consignments that have a very strong relation, for example because one consignment is succeeding the other (the goods are split, like the earlier examples). On the other hand, the consignments within a TransportOrder do not have to be related at all, they are simply 'grouped', which is similar but there is a subtle difference. edit looking at the above examples, consignments 1, 3, and 4 are strongly related, but consignment 2 is not strongly related to the others.

That said, I am sympathetic to your point of view. Because the ultimate goal is to make processes standardized and avoid confusion. If parties are going to use OTM5 differently, the benefit of using a standard is diminished.

I think that there is still a need to simply group things together without creating a strong connection, and thus that the TransportOrder needs to keep existing, even with the proposed change of relatedConsignments. That said, the more OTM5 grows the more the need arises to get clear rules about how OTM5 should be used. For example, currently it is indeed not clear where actors belong, on the Consignment or on the TransportOrders? Indeed, the general rule of thumb was to have the shared actors on TransportOrders (e.g. the issuer of the consignments) and the specific details in the Consignments (e.g. the exact location where each consignment should go to). But I think we should become more clear, potentially at the cost of being more verbose.

So before we continue with this approach it might be good to make those rules upfront and document before introducing a new concept. Though I think we need some more time to think about that (any suggestions for rules are welcome).

[BobZuidhoek]

Hi Bas,

You probably can guess my opinion on this already :-).

I think it is in this case indeed best to remove the 'duplicate' layer Actors and Constraint from Transportorder. And just maintain everything directly in Consignment. This is then 100% clear for everyone, prevents having to check/solve for inevitable inconsistencies between TransportOrder and Consignment and saves having to build additional parent/child logic into software.

Can you check with the DALTI community how TransportOrders are being used nowadays? To see whether there is some actual valid business reasons why you want to use the TransportOrder layer? For example, invoice grouping would not be a strong argument in my opinion. To my best knowledge, all roadfreight carriers want to get rid of complex invoice summary logic their shippers ask them to perform. As long as the consignment references are maintained, shippers should be able to effectively marry the carriers invoice lines to their own freight purchase order IDs.

If you would in the end conclude that TransportOrder is just 'legacy' logic to mimic the situation where shippers could send 1 large XML file with many consignments in it (for network efficiency), then keeping that legacy alive goes against the principle of REST. The basic REST-situation should remain that 1 TransportOrder/Consignment (if we merge these concepts into 1) = 1 API-call. This could lead to many 100's or 1000's of API-calls, but that should still be manageable. If there would still be a real need for batched API-operations, I think the TransportOrder/consingments in the batch then still each need to be processed in isolation (i.e. we should be wary of logic like "if 1 object fails then the whole batch must fail"). This is a bit of an odd website https://diegosucaria.info/handling-batch-operations-with-rest-apis/, but I'm charmed with how the paragraph Create a new batch endpoint and then particularly the idea of POST /batch/users works. The idea of having dedicated API endpoints for batch operations makes sense to me (and keeps the solution clean from the regular true REST endpoints). The example should be improved with a proper batch-ID and ID's for each line in the array, but you get the point.

[bmeesters]

Can you check with the DALTI community how TransportOrders are being used nowadays?

We started out with a beta version of OTM5 that did not have the TransportOrder, and almost all parties involved (with many representatives in DALTI) explicitly wanted to include it. So including it seems like quite a big boon for OTM adaption. Whether or not this is a valid enough reason from a technical point of view might be questionable, but I think removing it now would be 'political suicide'. OTM is already somewhat controversial with certain decisions that you can argue are better, but still seem to hinder adoption (see also https://github.com/opentripmodel/otm5-change-requests/issues/29). We can discourage it's use maybe, but I don't think we can remove it. Integrating in current work processes is essential for adoption, even if these processes can be somewhat outdated.

I think it is in this case indeed best to remove the 'duplicate' layer Actors and Constraint from Transportorder.

We cannot remove fields without bumping the OTM version to 6. But we can deprecate and discourage it until the time is right to make the next jump. In any case should we provide more clear rules on how to use these constructs.

I made a 'change request' for the documentation regarding the subject https://github.com/opentripmodel/otm5-change-requests/issues/34. Because how to deal with TransportOrder and Consignment actors and constraints is in a sense an orthogonal concern to whether or not to add relatedConsignments.

To be clear (what might get lost in conveying over text), I do really appreciate the critical questions and suggestions. This is what helps OTM5 improve and grow over time. So thanks @BobZuidhoek and please continue doing so!

[thomaskolmans]

Over the weekend I have thought about this but I would very much urge to keep TransportOrder in place, even with the related Consignments. For administration purposes and ease-of-use I believe that having a base-line of TransportOrder will simplify and unify a lot of the data being transferred. The structures that can and will be created if a Consignment is also the TransportOrder with relatedConsignments.

In version 4 of OTM it was this way and created quite a few complexities these where taken away with the TransportOrder.

[BobZuidhoek]

Hi Thomas,

For knowledge sharing, can you share what typical functional cases are now being solved with the TransportOrder in place? I have the idea that roadfreight carriers do things with this concept that are worth knowing about.

BTW, I'm certainly not challenging the use of TransportOrder being there. I'm just trying to understand what sensible arguments we may run into when my company (=shippers that want to integrate with all Dutch roadfreight carriers via OTM5) would integrate with each of these carriers TMS's.

[thomaskolmans]

As a TransportOrder doesn't do anything as in comparison of the Consignment. It get's used as an administrative reference and a baseline for everything else. Doesn't have anything to do with roadFreight on it's own, we're using OTM for multimodal solutions, but see it as the foundation of the administrative data exchange.

Once of your, in some perspective very understandable, argumentation that the TransportOrder status relies on the underlying Consignments. Well yes, that's true, but now you at least will know what is all underlying. As you may be consolidating consignments, changing certain relatedConsignments to a different modality or a different Route. It becomes every more complex which has what status if you don't have that foundation.

Now you at least know what the baseline is and you can, recursively if need be, asses the status. That has to happen in any complex case in any way. Hopefully that makes sense.

Beyond a direct OTM argument, we use it for our entire administrative and integration foundation too, as it's the only "non-moveable" item that relates to a client and the "moveable" things. This means that it's stable and a good object to attach all administrative and further processes too.

If you have any further questions, I'll gladly expand on my answer or explain more from where we are coming.

[BobZuidhoek]

Hi Thomas,

I've worked in the forwarding industry for 6 years (this is not my current job by the way), dealing with multimodal transport of aifreight, seafreight and roadfreight (no trains or inland water). So I do understand pretty well how consignments get consolidated / deconsolidated to travel with one another as part of total transportation execution route.

Couple of questions then come to mind:

1. Do you really use OTM5 to also describe things such as an airfreight or seafreight transportation leg? There are pretty different things going on there (MAWB/HAWB airwaybills, flights, seawaybills, etc). The scope of OTM5 really is just roadfreight, or am I missing something? This is also how its mainly organized (e.g. DALTI, SUTC, etc.).
2. Are you interpreting a TransportOrder as some form of functional consolidation? Or are you really just using the TransportOrder to send a batch of (related and/or unrelated) consignments to a roadfreight carrier?
3. If you're talking about a roadfreight leg (could be 1 or more roadfreight legs) within the total multimodal transportation process, won't you have your master-references in your own internal systems?
4. Is the carrier expected to always refer back to the TransportOrder references somewhere (e.g. invoicing).

[thomaskolmans]

1. Do you really use OTM5 to also describe things such as an airfreight or seafreight transportation leg? There are pretty different things going on there (MAWB/HAWB airwaybills, flights, seawaybills, etc). The scope of OTM5 really is just roadfreight, or am I missing something? This is also how its mainly organized (e.g. DALTI, SUTC, etc.).

Yes the bills are different and we've had to expand it to some extend of course, but the base principles are the same. Thus we've built upon the OTM model.

2. Are you interpreting a TransportOrder as some form of functional consolidation? Or are you really just using the TransportOrder to send a batch of (related and/or unrelated) consignments to a roadfreight carrier?

Not as an internal consolidation but it is the "package" that everything starts from. It's what the client sends to the system, that is how we interpret it.

3. If you're talking about a roadfreight leg (could be 1 or more roadfreight legs) within the total multimodal transportation process, won't you have your master-references in your own internal systems?

We've got a master reference, that is once again a TransportOrder that will be then divided into multiple Trips etc.

4. Is the carrier expected to always refer back to the TransportOrder references somewhere (e.g. invoicing).

Yes exactly, not just invoicing, in backward integration (communicating location, load, status, etc.) this is being used as a reference.

In essence, we interpret the TransportOrder as the "order" that is being put in. Whatever is in that order will be moved, but the Order itself is static and can be the reference point towards the order-giver (e.i. the client)

[BobZuidhoek]

OK thanks for that. That's clear.

Do I understand correctly that your using the OTM domain model as very close inspiration (reference) on how to design your software (is that a TMS or other system)? That might explain the context better of what you're doing (and why you're quite easily adapting or expanding the OTM5 domain model within your software).

The way I look at OTM5 is that it is rather a shared domain model to function as a foundation for system integration (messages between systems). That usually implies that the systems themselves may have more complex or even very different domain models within them. I think it is important that we should keep the focus on the structure of data flows between systems, instead how trying to model OTM5 as close to how that could be implemented in actual software. For example, if a client (shipper) books a shipment and you've modeled that booking in your TMS as a Transportorder, it may not necessarily mean that OTM5 TransportOrder is also used in the same context when sharing data between systems.

At least that's my take on it...

All in all, the higher the need for OTM to become more clear on what is actually meant with the TransportOrder concept. Maybe from this point it's best to wait for Bas to create some reference documentation on this. That already is an action point in a different Github issue.

[thomaskolmans]

Yes I do know that and yes we've based our TMS on OTM, so I am speaking from that perspective. However, I have worked wherein data standards are already widely implemented. In order to easily maintain a data standard and to easily communicate, it should be able to adhere to general concepts within dataflows.

Even in a perspective of using it as a communication layer I still believe my argumentation holds true. You'll have to communicating some way all possibilities. I believe if you let go of TransportOrder, you lose the base reference point in my opinion. Also in "just" transferring data.

[bmeesters]

Great to see this open discussion. If I need to update any information on the documentation (either the developer portal or the API docs) just let me know.

For now, the overall change (related consignments) has been accepted for OTM5.3

[etwobben]

hi @bmeesters ! I would like to implement this feature request it advance as we kinda need it now :-) Is there already an consensus on how the model is going to look like? As far as I know there was some doubt about the relationfield in an asscocation.

[bmeesters]

Hello @etwobben I have to admit I forgot a bit about the relation field. Though, now having given a bit more thought again I think it makes sense to add it. For a lot of associations the relation is pretty obvious (the actions below a trip, the vehicle on the trip, the consignment under a load or unload, etc.). However for relatedConsignments (like actors) it might not be obvious (e.g. which one is supersedes which and why).

However I am not sure what would be the best relation types to add (initially). Do you need the relations field and do you have any suggestions that need to be there?

[etwobben]

The relation is something we need to describe the reason why something is related. For now I can only think about these;

• split, for consigments that are oversized for example.
• postponed. this is for consigments who can't be delivered (partially delivered) and a new consigment is created with a new delivery date.

The descriptionfield in a association can be used for some more information.

[bmeesters]

Sounds good to me. We can always add more relation types in future OTM versions if the need arises.

edit I wonder though if we want relations to go one way (e.g. the old one references the new one, but not the other way around) or both ways. I can imagine both are useful, but it might complicate things and we would also need more types to model the reverse relationship).

Any thoughts on this @etwobben?

[bmeesters]

This has been implemented in OTM5.3 released today and thus can be closed.