usdot-jpo-ode / jpo-geojsonconverter

3 stars 6 forks source link

Partition outputs by Intersection ID #36

Closed iyourshaw closed 1 year ago

iyourshaw commented 1 year ago

To support scalable deployments of the CIMMS in regions where there is not a one-to-one correspondence between origin RSU IP and Intersection ID for MAPs, SPATs, and BSMs, we need to change the Kafka partitioning scheme of the output topics for ProcessdMaps and ProcessedSpats to be based on the Intersection ID instead of RSU IP address. For example, we have observed in some deployments that BSMs can originate from a different RSU than the matching MAPs and SPATs. To maximize the efficiency of joins, and to avoid having to create additional topics, it would be preferable for the geojson converter to produce MAPs and SPATs pre-partitioned by intersection ID, rather than having to do the repartitioning within the CIMMS.