Open ryankurte opened 7 years ago
guyharris
commented
7 years ago Supporting GPS Location and frequency in the IDB or EPB allows for specification of multiple static capture points or moving captures at different frequencies.
So presumably if the capture device isn't moving, the location can be in the IDB rather than in every EDB, and if it's not doing scanning, the frequency information can be in the IDB rather than in every EDB.
Would the same apply to the channel number?
guyharris
commented
7 years ago RSSI in dBm as 1x 32-bit float for EPB
What if you're receiving on more than one antenna?
guyharris
commented
7 years ago Channel as 1x 16-bit uint for EPB
What indicates how the channel number is to be interpreted? I.e., is this an 802.11 channel, a Bluetooth channel, etc.?
ryankurte
commented
7 years ago Thanks for the quick response! In order:
WRT channel number in EPB or IDB, yep that would make sense.
WRT RSSI, I'm not aware of hardware that actually receives the same packet on N antennae rather than just picking the best one at a given point in the preamble, but, support for multiple RSSI options per packet would work? (And if they're actually parallel, that may be better modelled as N interfaces).
WRT. channel number interpretation, this is I guess backwards from the current way. Rather than a protocol specific header (ie. radiotap) that encodes the channel and how to interpret it, pcap-ng can encode the raw channel, then the decoder can handle interpretation if it's required (along with everything else in the pcap payload).
That does raise the question of support for further metadata (like channel b/w, spacing), as it stands I have been putting that kind of data in the file and interface comments since it's not /required/ for parsing / understanding the packet capture.
kismetwireless
commented
7 years ago I would also be interested in GPS data; I'd propose pushing it to include some additional info, modelled on the PPI GPS framework:
https://media.blackhat.com/bh-us-11/Cache/BH_US_11_Cache_PPI-Geolocation_WP.pdf
The lat/lon/alt/ts is very similar to what is proposed above; the addition of VECTOR would allow for precision aimed antennas (something the original ppi geolcoation method was designed for).
Position in both IDB and EDB would account for all my needs (fixed and moving sensors); IDB alone wouldn't be enough.
I'd like to suggest breaking the record into 'gps' and 'wifi specific'; I've got non-wifi packet sources in Kismet which still have location data.
For the wifi-specific component, you may find it difficult to represent channel simply; in wifi alone you've got 5, 10 15 (non-standard) widths, 20, HT20, HT40+ HT40-, HT80+center, HT160+center, and HT80+80 with dual non-contiguous center frequencies.
For RSSI, any N/AC hardware with mimo will report multiple signal levels in the radiotap header; you've got an arbitrary number of antenna signal levels (1-3 on current hardware but more in the future is certainly a possibility).
ryankurte
commented
7 years ago @kismetwireless good points! IDK whether the pcapng team would rather one big issue or multiple smaller ones (or how to actually enact any of this 😂)?
Totally agreed about position options for both IDB and EDB.
So the VECTOR addition would add pitch/roll/yaw? I would make the suggestion that perhaps that would better be called orientation, and I have been wondering whether speed was useful to encompass doppler shift of measurements. Also that hopefully it could be more simple.
I don't understand what you mean by 'gps' and 'wifi specific'? The goal here is to have useful physical data available for any packet capture source (I work on a variety of ISM band mesh networks which is the use case for me)
That is true, but the question is whether pcap-ng should include that, or, have some kind of generic enough mapping for it to be implemented for arbitrary protocols.
I would argue the latter, but we have to find a way of generically representing it.
Perhaps the addition of a channel name option would be sufficient?
Another option would be to add channels to the IDB and reference them from each EPB, but that's adding quite a lot of complexity, and means that wifi channels will end up getting encoded in every pcap file.
For RSSI, would the addition of an antenna index to the rssi option cover that?
ryankurte
commented
7 years ago RSSI could be 2x 16-bit to save a 4 bytes, but, 32-bit floats are probably easier to deal with in most situations (unless you're missing an FPU).
kismetwireless
commented
7 years ago re: wifi vs gps - we need gps for non-wifi packets, so it doesn't make sense to couple them. We should have a GPS EPB and IDB, and a 'frequency' or similarly named EPB and IDB.
For the GPS frame I'd suggest all the data mapped in the GPS PPI spec which I linked above, starting on page 22.
guyharris
commented
6 years ago See also another proposal for a "Location Information Block" in the archives of the pcapng mailing list. Here are the thread from March 2015 and the continuation of that thread in April 2015.
ryankurte
commented
6 years ago @kismetwireless absolutely and that's why I am proposing this? AFAIK it's currently coupled because the recommended way of getting location data into wireshark is wrapping whatever data you have in a radiotap header? And yep, the PPI spec seems like a good basis, which it seems to me would be a small addition to the previous discussions linked by @guyharris.
@guyharris thanks for linking those, do you know what happened following that discussion? Also, how to develop this discussion into a useful submission? It seems like we could pick up a lot of the previous components without much modification, though I would argue for floating point and sane units if that's appropriate within the spec.
(For other readers, this post contains a fairly thorough formatted proposal with responses following, and this covers many of the same options as we're talking about with different formats/units (with much of the PPI spec stuff).
I think the distinction between a location option and a gps option is important, and I used the wrong nomenclature before, the use case for me is to know from where the packet was observed (or created) regardless of the mechanism providing the location information.
Is the underlying mechanism relevant in the capture file, and is the proposed Location Provider Block worth the complexity of having to deal with indexing as opposed to introducing additional option blocks?
guyharris
commented
6 years ago do you know what happened following that discussion?
There were still questions outstanding, so nothing happened after that.
Also, how to develop this discussion into a useful submission? It seems like we could pick up a lot of the previous components without much modification,
"This discussion" meaning the discussion here on GitHub, or the e-mail discussion from 2015? The best thing to do with the 2015 discussion is to pick up from the proposal and answers to questions any ideas that look good and, for open questions, possibly resolve them (e.g., the questions about the Source Types) and incorporate the results into the your proposal if they look good.
I think the distinction between a
locationoption and agpsoption is important
As in "we need to offer both types of option" or as in "the option should be called "location" rather than "gps", as it might not come from GPS or even from any of the satellite-based positioning systems"?
and I used the wrong nomenclature before, the use case for me is to know from where the packet was observed (or created) regardless of the mechanism providing the location information.
So it sounds as if you mean ""the option should be called "location" rather than "gps""; if so, I agree.
Is the underlying mechanism relevant in the capture file
I'll leave that for others to answer.
is the proposed Location Provider Block worth the complexity of having to deal with indexing as opposed to introducing additional option blocks?
Options in the IDB, like options in the LPB, can be thought of a a form of compression - instead of having to put values for those options in every EPB, if they'll be the same for every packet on a given interface, or for every packet geotagged by a given location provider, there can be one instance of them in the IDB or LPB and the EPB can just point to the IDB or LPB by ID. You already have to do indexing to refer to the IDB to get, for example, the link-layer header type; similar code an do indexing to refer to an LPB, so I don't consider it an extreme amount of extra complexity.
I guess the "compressed" options could be attached to an IDB, but if you have, for example, a laptop with two WI-Fi adapters and a GPS receiver, you have one location provider and two interfaces, so you conceptually have one location provider, so I could see having an LPB as being a bit cleaner in that sense.
guyharris
commented
6 years ago AFAIK it's currently coupled because the recommended way of getting location data into wireshark is wrapping whatever data you have in a radiotap header?
There are two ways to provide per-packet metadata: provide them in a link-layer-header-type-specific metadata header preceding the link-layer header, or provide them in pcapng options.
The first one has the advantages that:
they don't require pcapng - they can be used in pcap files as well;
if they're specific to a particular link-layer type, you don't have to deal with them in inappropriate link-layer types (so, for example, your code to handle LINKTYPE_ETHERNET doesn't have to figure out what to do with an Ethernet frame tagged with the number of spatial streams - display it or ignore it?)
The second one has the advantage that, if it's not specific to a particular link-layer type, you don't have to add it to N different link-layer types.
The location/angle/velocity information is definitely not link-layer specific, and strikes me as an obvious candidate for inclusion in pcapng (at the expense of it not being representable in pcap, but there's already information that's not representable in pcap).
ryankurte
commented
6 years ago @guyharris thanks for the responses! The discussion being all of the discussions in all the places ^_^
incorporate the results into the your proposal if they look good
Cool, and what format should the proposal be? Would a PR against the base XML file work?
the option should be called "location" rather than "gps"
Exactly.
Options in the IDB, like options in the LPB, can be thought of a a form of compression - instead of having to put values for those options in every EPB, if they'll be the same for every packet on a given interface, or for every packet geotagged by a given location provider, there can be one instance of them in the IDB or LPB and the EPB can just point to the IDB or LPB by ID. You already have to do indexing to refer to the IDB to get, for example, the link-layer header type; similar code an do indexing to refer to an LPB, so I don't consider it an extreme amount of extra complexity.
IDK how much more it is for parsers, but, my suggestion is that putting these in the IDB and EPB is still simpler compared to introducing an LPB as the IDB lookup already exists.
As mentioned in the mailinglist adding the LPB is also going to increase the complexity of merging captures, whereas the other approach should work as without modification to existing tools.
I guess the "compressed" options could be attached to an IDB, but if you have, for example, a laptop with two WI-Fi adapters and a GPS receiver, you have one location provider and two interfaces, so you conceptually have one location provider, so I could see having an LPB as being a bit cleaner in that sense.
It is probably cleaner, but for N adaptors in a static (not moving) configuration at the same location, in which case it's going to save N-1 location headers. The other case is moving with N adaptors at the same location, in which case if packet rate >> location update rate it could save some, which might be more worthwhile. In any other case, ie. different locations, packet rate << location update rate, there is going to be a ~1:1 correlation between EPBs and LPBs which will be like anti-compression.
IMO it would be a mistake to end up allowing both, so, the to conclude this the options are either:
The former will (probably) result in better compression for some use cases, but requires modification of existing tools to support merging (and parsing?) pcap-ng files containing LPBs.
The latter may result in data duplication where N > 1 interfaces are used from the same place under certain conditions, but should not require any modification to support merging or splitting of files. IMO it's also easier to implement for parsing / encoding, but, that doesn't matter a lot.
That seems like a question of ethos as much as anything?
The location/angle/velocity information is definitely not link-layer specific, and strikes me as an obvious candidate for inclusion in pcapng (at the expense of it not being representable in pcap, but there's already information that's not representable in pcap).
That's why we're here ^_^ I couldn't quite stomach the thought of wrapping my 6lowpan packets in radiotap headers / writing more oddly specific encoders.
guyharris
commented
6 years ago Cool, and what format should the proposal be? Would a PR against the base XML file work?
If there's a convenient way of formatting the updated XML file into the resulting HTML and viewing it, before the chances are merged, yes. Not being able to preview the result would make it more difficult to comment on.
guyharris
commented
6 years ago The other case is moving with N adaptors at the same location, in which case if packet rate >> location update rate it could save some, which might be more worthwhile. In any other case, ie. different locations, packet rate << location update rate, there is going to be a ~1:1 correlation between EPBs and LPBs which will be like anti-compression.
I think there's some confusion here. Kenneth D. Renard's original proposal had a Location Information Block (LIB), which provides location information. I later proposed a Location Provider Information Block (LPIB) which gives a description of a provider of location information.
A capture would have one LPIB per source of location information, no matter how many interfaces or packets there are, or how fast the capturing machine moves.
The location update rate would control the number of LIBs, not the number of LPIBs.
So there are at least two questions:
If we provide a description of the source of location information, should we do so by putting the description in an LPIB or in an IDB?
Should location information be attached to an EPB or an LIB?
I was mainly discussing the LPIB vs. IDB issue; you appear to be discussing the EPB vs. LIB issue.
For LPIB vs. IDB, as per my comments, I see having a separate LPIB as conceptually cleaner, even if it requires that the reader keep track of lists of location providers as well as lists of interfaces.
For location information in EPBs vs. location information in LIBs, that's where the packet rate and location update rate matter. In the thread about Renard's proposal, I proposed per-EPB location information in this message:
If somebody's capturing while in motion, and we're not interested in position or orientation information when packets aren't arriving, we could have those options available for both LIBs and packet blocks, and say that the position/orientation information for a packet is unchanged from whatever the last values were for that interface if the options aren't present. That might allow less stuff to be written if we're only interested in the position and orientation at the time packets are captured.
and Renard responds in this message:
Agreed. Frequency of location information is defined by the application/user. Example use cases might be:
- Set location description each time I wake up my laptop
- Synchronous stream from GPS: Once per second, per interface.
- Every N seconds or change in position more than M meters.
Location information per-packet seems extreme, but certainly valid.
I see the LIB as useful only if either 1) the machine is moving slowly enough, and capturing enough traffic, that you'd get less redundant location information (that's the packet rate >> location update case you mention) or 2) the user wants to know how the machine is moving even if it doesn't happen to be capturing any packets (that's a packet rate << location update rate case, but it's not necessarily all packet rate << location update rate cases - if the user doesn't care where the machine is when it's not seeing traffic, the extra LIBs aren't useful). If the two rates are comparable, the LIB doesn't provide any compression.
ryankurte
commented
6 years ago I think there's some confusion here. Kenneth D. Renard's original proposal had a Location Information Block (LIB), which provides location information. I later proposed a Location Provider Information Block (LPIB) which gives a description of a provider of location information.
Yep, I somehow got the two of them mashed up in my head. s/LPB/LIB/g in my previous comment, and thanks for clearing that up!
If we provide a description of the source of location information, should we do so by putting the description in an LPIB or in an IDB?
IMO the mechanism of location capture is metadata that isn't particularly relevant to the captured packets, though it would make pcap files more standalone.
I see the LIB as useful only if either 1) the machine is moving slowly enough, and capturing enough traffic, that you'd get less redundant location information (that's the packet rate >> location update case you mention)
I suspect given enough interfaces it's probably pretty easy to receive >> packets than location updates (GPS is usually ~10Hz which is /not very fast/).
2) the user wants to know how the machine is moving even if it doesn't happen to be capturing any packets
I don't think this is a use case for a pcap-ng file? If you want a GPS track you can save one using a format for that. That said, if we're implementing a location information block (which has an interface as a parent) it's not a big deal to support it.
I need to map out the implications of each approach again but it's way too late here. Will try and tabulate it tomorrow.
If there's a convenient way of formatting the updated XML file into the resulting HTML and viewing it, before the chances are merged, yes.
It's gotta be achievable, convenience though 😂 Will have a look sometime this week into what can be done.
taospartan
commented
6 years ago Hi Guys, Has there been any movement on this feature?
cheers
tao
ryankurte
commented
6 years ago Not yet, still need it / intending to work on it but buried under other things at the moment.
ryankurte
commented
6 years ago After a long time of not having the time to think about this I had the idea that perhaps rather than building this to be location specific, it would make more sense to have a Capture Information Block (CIB) that we can extend with options for location or whatever else is required, and attach to interface description block(s) or refer to from packet block(s).
So, a draft of that idea is in progress in #51. It doesn't address all the questions here or in the mailing list yet (because tbqh I'm finding keeping track of all the discussion rather difficult), but, we'll get there hopefully.
The immediate questions I have are:
Capture Information Block make sense?epb_cib_id option in EPBs a sensible way of linking to packets?cid_interface_id option in CIBs a sensible way of linking to interfaces?Location Provider Information Block (LPIB) equivalent?
What do y'all think?
guyharris
commented
6 years ago It doesn't address all the questions here or in the mailing list yet (because tbqh I'm finding keeping track of all the discussion rather difficult)
So we now have discussion on:
It might be easier to keep track of it if it's all in one place; if so, where should that be?
ryankurte
commented
6 years ago I think it would make sense to continue here?
guyharris
commented
6 years ago I think it would make sense to continue here?
I'm fine with that so, unless somebody else objects, let's do that (and not further discuss this proposal on the mailing list or in comments on the pull request).
guyharris
commented
6 years ago Does having a generic Capture Information Block make sense?
So what does "capture information" refer to? It's presumably not information about a particular packet, because that should be done as options for packet blocks of various sorts (Enhanced Packet Blocks or any future packet block types).
So it could be thought of as a form of compression for cases where the same information applies to multiple packets, under the assumption that the information in question changes rarely enough that you save space by not duplicating the information in multiple EPBs.
The current proposal has only position, orientation, and velocity information; presumably those change slowly enough that you would get sufficient compression by using CIBs (and so that, in a capture taken over a very long period of time, you don't get more than 2^32 changes of position, orientation, or velocity!).
Do you foresee other forms of information where this putting the information in a CIB, rather than in per-packet options, would result in compression rather than expansion?
guyharris
commented
6 years ago
- What is missing in the CIB?
- Timestamps
What sort of time stamps? A time stamp of the time at which the information was determined?
- RF info (though this needs more discussion)
Does that change infrequently enough that putting it in a CIB rather than in per-packet options would result in compression rather than expansion? Does the answer depend on which particular RF information is being discussed?
guyharris
commented
6 years ago Does attaching these to IDBs and EPBs cover our intended use cases?
If the intent is to provide compression of per-packet information that changes sufficiently infrequently, attaching them to EPBs would cover that use case. Attaching them to an IDB would presumably be useful for the case where the information never changes, e.g. a capture done on a machine that's not moving relative to the surface of the earth could have a CIB giving the position, orientation, and velocity of the device (the latter presumably being the zero vector, so maybe it needn't even be reported) attached to all of the IDBs for the capture.
BTW, if an EPB has a CIB attached to it and is attached to an IPB with a CIB attached to it, if a given option is present in both CIBs, should the option value in the CIB attached to the EPB be the one used?
guyharris
commented
6 years ago Do we still need a Location Provider Information Block (LPIB) equivalent? I'm not convinced it offers a whole lot of utility if CIBs can be attached to IDBs or EPBs?
So presumably you're proposing CIB options that could describe a location provider. How often would that information change? If it never changes, putting that in a CIB attached to the IDB would work, but if it can change occasionally, but changes more slowly than other CIB information, it'd be duplicated in all CIBs starting with the first change from values specified in the CIB attached to the IDB.
A quick look at Wikipedia suggests that not all satellite-based navigation systems have global coverage yet, so it might be possible for a mobile device capturing packets to switch from one provider to another if the device moves out of the area covered by the nav system it's currently using. By 2020 that might not be a problem (I don't know whether there would be other reasons, technical or political :-), for a device to change from one nav system with global coverage to another while in the process of doing a capture).
guyharris
commented
6 years ago Given its absence from the proposal, presumably angular velocity (d{orientation}/dt) isn't of interest?
guyharris
commented
6 years ago I'm fine with that so, unless somebody else objects, let's do that (and not further discuss this proposal on the mailing list or in comments on the pull request).
Except for obvious editorial changes (typoes, copy-and-pasteos, etc.), which are probably best done in comments on the pull request.
ryankurte
commented
6 years ago Thanks for all the commends and feedback! On working through them I think there's a simpler approach, presented first so it's more obvious it's here.
I can update the PR to reflect these changes if you like to see them, or wait until they've been further discussed.
So what does "capture information" refer to? It's presumably not information about a particular packet, because that should be done as options for packet blocks of various sorts (Enhanced Packet Blocks or any future packet block types).
It could be called "Capture Metadata Block (CMB)" or anything similar, I am open to alternate names if there is a more appropriate one. My original proposal was just to add options to the Enhanced Packet Block (EPB) or Interface Descriptor Block (IDB), however, I think it makes sense to use a separate block:
Do you foresee other forms of information where this putting the information in a CIB, rather than in per-packet options, would result in compression rather than expansion?
Definitely more location-related ones, I'm not sure about other types of information. Some RF measurements (like RSSI) are definitely better per packet, some like channel bandwidth are per-interface and/or change with time, but this name supports it if we come up with other options.
What sort of time stamps? A time stamp of the time at which the information was determined?
Yeah, if we're using global lat/lng/alt from a GPS it'll probably be nice to have the GPS time along with it.
Does the answer depend on which particular RF information is being discussed?
Yep. Bandwidth / Frequency / Channel spacing are often per-interface for simple networks, but are not for wifi etc (as @kismetwireless pointed out). If you have a channel hopping interface and it receives > 1 packet per channel hop it might be useful to define the channel there. Except, you can also have wide-band interfaces listening on multiple channels at once, which is really only served by per-packet information. Also, it makes no sense having something like RSSI in a reusable block because it only exists per packet.
If the intent is to provide compression of per-packet information that changes sufficiently infrequently, attaching them to EPBs would cover that use case. Attaching them to an IDB would presumably be useful for the case where the information never changes.
I was thinking to link the information blocks to the interface blocks (rather than vice versa), so you can re-issue updates to an interface, and the latest CIB applies to all packets on that interface until the next one is read.
This also works kinda nicely from the writing side, if you're capturing packets you just have to interleave CIBs when you receive updated location information and they'll apply to everything until your next CIB, which also works nicely for the static case.
It does mean if you have multiple interfaces in the same place, you need multiple CIBs. We could allow them to be attached to N interfaces if that was desired, though it seems a little more complicated.
position, orientation, and velocity of the device (the latter presumably being the zero vector, so maybe it needn't even be reported).
The CIB is really just a list of options, so whatever is relevant can be reported and whatever is not can be elided.
BTW, if an EPB has a CIB attached to it and is attached to an IPB with a CIB attached to it, if a given option is present in both CIBs, should the option value in the CIB attached to the EPB be the one used?
The EPB attached block would override and of the same options specified in the IDB attached block, so the EPB version lets us override the IDB version.
On thinking more I'm not completely sure we need to be able to attach them to EPB at all, it might be better to rip out the strictly per-packet options like RSSI and channels as EPB options and remove the ability to link this with an EPB, which would vastly simplify the complexity of parsing logic.
it might be possible for a mobile device capturing packets to switch from one provider to another if the device moves out of the area covered by the nav system it's currently using
In my experience, most multi-constellation systems still output lat/lng/alt, I don't think which system is providing the information is necessarily relevant. We could add an enum to support that identification, but, I don't know how much sense that makes when you're using a variety of networks.
I'm not convinced this is a problem that should be solved within pcap.
Given its absence from the proposal, presumably angular velocity (d{orientation}/dt) isn't of interest?
I'm not interested in it, but, once we have decided how we're going to do this / split up interface or packet related options, introducing new options is reasonably trivial.
guyharris
commented
6 years ago So, for the local location information, is there a need to specify where the origin is?
guyharris
commented
6 years ago My original proposal was just to add options to the Enhanced Packet Block (EPB) or Interface Descriptor Block (IDB), however, I think it makes sense to use a separate block:
- for compression as you mention
- to allow the metadata to be reused against EPBs / IDBs (/ any future packet blocks)
"Reused" as in "not having to separately define IDB and EPB options"?
- so we have a mechanism to update metadata through time
If the metadata is per-packet, then there's nothing to update, so this appears to refer to updating values specified in an IDB.
guyharris
commented
6 years ago In my experience, most multi-constellation systems still output lat/lng/alt, I don't think which system is providing the information is necessarily relevant. We could add an enum to support that identification, but, I don't know how much sense that makes when you're using a variety of networks.
I'm not convinced this is a problem that should be solved within pcap.
So let's not bother with providing a mechanism to identify the location provider until somebody requests it and gives a good argument why it's necessary.
ryankurte
commented
6 years ago So, for the local location information, is there a need to specify where the origin is?
The specific scenario I am thinking of is the arbitrary grid space of RF simulators, so, no I don't think so.
"Reused" as in "not having to separately define IDB and EPB options"?
Yeah, though I think the alternative here is a simpler approach.
If the metadata is per-packet, then there's nothing to update, so this appears to refer to updating values specified in an IDB.
Not specified in, but attached to?
So let's not bother with providing a mechanism to identify the location provider until somebody requests it and gives a good argument why it's necessary.
Agreed.
This would seem to be the simplest option, though does require all metadata to be repeated every packet.
An example file would be:
It's very easy, though not super efficient and does not support location sampling >> packet receipt rate.
Create a metadata block and associated options that are linked to a given packet using an option in the EPB.
This would look something like:
This requires some ordering rules (metadata defined before packet so the data is available when streaming), and tracking of metadata block IDs. The static case requires only one metadata block (per interface), however, packet block sizes will increase slightly with metadata link information.
I don't think you can link the other way without introducing unique EPB IDs or an EPB ID counter, which generalises poorly for other packet types.
For persistent metadata (eg. location), create a metadata block that is always applied to an interface and applies to any packets following it. For packet-related metadata (eg. RSSI), add options to EPBs. This is what I proposed here, but, I may not have been clear enough about it.
An example file might look like:
This allows compression for the static use case (eg. where location doesn't change) and where packet rate >> change in metadata rate. Where this is not true, up to one metadata block may be required per packet block. Persistent metadata is also generalized over all packet types, and interface location can be tracked at a higher rate than packet capture if required.
My preference is for option three as I think it's reasonably simple/efficient/covers all use cases, or option one for simplicity, and I would be happy to go ahead and rework the PR to match either. Am I missing anything / are there any other approaches we should still be considering?
ryankurte
commented
6 years ago @guyharris any thoughts on these updates?
packetfoo
commented
6 years ago Sorry, I was out of the discussion for a while for various reasons, including not being very familiar with all the WiFi stuff (which started the discussion).
I think option three is the best, even though it adds the most complexity.
Do I read the proposal correctly when I assume that MDBs are required when we introduce them (always having at least one MDB per IDB?) I haven't completely made up my mind about it but it may be desirable to still be able to write pcapng files with no MDBs at all. In which case the linking needs to be done via block options, which I would prefer anyway (compared to adding static link fields in the block headers).
ryankurte
commented
6 years ago For option three, my suggestion is that the MDB includes the link to the interface it applies to (ie. contains the interface id), and if MDBs are available and the parser supports it, metadata gets applied, otherwise they're just another opaque block. That way there should be no compatibility issues (, and you can simply strip or merge pcap files and metadata if desired).
packetfoo
commented
6 years ago Yes, that sounds like the right way to do it.
ryankurte
commented
6 years ago 
b1tninja
commented
6 years ago Please implement?
ryankurte
commented
6 years ago @packetfoo did you have a chance to look at #51? I guess the things left are:
edit: todo list moved to #51
packetfoo
commented
6 years ago @packetfoo did you have a chance to look at #51? I guess the things left are:
* [ ] Should we call this `Capture Information Block (CIB)` or `MetaData Block (MDB)`? * [ ] Examples (do we need them?)
No, I haven't, but I'll look at it this week in greater detail. I'm fighting with git and Github (the workflow concepts of Github refuse to get into my head, apparently), but I'll get there eventually...
ryankurte
commented
6 years ago Cool cool cool, if it'd be useful we could run through it in a skype call or something, just let me know if I can help (my email is my username at gmail.com if you'd like to get in contact that way) ^_^
jkcko
commented
5 years ago May I ask what happened to the proposed EPB RF options (RSSI, channel, freq, etc)? The PR #51 only addresses the location information in a CIB and the per packet options have fallen by the wayside.
I would like to also propose additional metadata for time slotted channel hopping radios which include an absolute slot number and start of slot timestamp (separate from start of frame).
ryankurte
commented
5 years ago May I ask what happened to the proposed EPB RF option
It turned out that there were a bunch of separate concerns with location information (which relates to interfaces) vs. rssi (which relates only to packets), so I decided to split the problem and attempt the location information in #51 and follow that up with another PR for the radio information.
As far as I can see, RSSI and Channel can just be added as fields on the EPB, which I plan to propose once this has landed.
jkcko
commented
5 years ago Since PR #51 is created for the CIB options shall discussion on the EPB RF options continue in this issue?
Some comments to those specific options.
Channel / Medium options a. Frequency in Hz as 1x 32-bit float for EPB and IDB b. Channel Spacing in Hz as 1x 32-bit float for IDB or EPB c. Channel Bandwidth in Hz as 1x 32-bit float for IDB or EPB d. Channel encoding as string for IDB or EPB e. Channel name as string for IDB or EPB
a. 32-bit is insufficient - even as an unsigned integer it wouldn't represent frequency above ~4.295GHz (i.e. 5GHz band), perhaps an option in CIB/IDB for the multiplier factor for frequency values? b. are there sub-Hz channel spacing implementations? Is 32-bit integer sufficient? c. is 32-bit integer sufficient? f. Data rate in bps or kbps as 32-bit integer for IDB/CIB, is a multiplier needed here as well?
Capture / Measurement options a. RSSI in dBm as 1x 32-bit antenna index and 1x 32-bit float for EPB b. Channel index as 1x 16-bit uint for EPB
a. separate antennas could also be handled as separate interfaces but perhaps not for MIMO configuration? So multiple instances of the RSSI option could appear in each EPB.
RSSI could be 2x 16-bit to save a 4 bytes, but, 32-bit floats are probably easier to deal with in most situations (unless you're missing an FPU).
I agree with the first part... Is RSSI as float needed? Do any chipsets report RSSI in values other than integers? Most situations not involving low powered embedded network devices without an FPU. I would prefer to see values represented as integers where possible.
Also add for TDMA and TSCH channel access methods.
kismetwireless
commented
5 years ago Re: per-antenna data and MIMO; we've already got 4 signal levels per packet for some wifi cards now and it'll only get more complex.
Right now, if I recall, radiotap handles this by appending additional capabilities blocks w/ only the antenna signal filled out; generating multiple records per packet in pcapng wouldn't be a problem.
On Thu, Dec 20, 2018 at 2:40 PM James Ko notifications@github.com wrote:
Since PR #51 https://github.com/pcapng/pcapng/pull/51 is created for the CIB options shall discussion on the EPB RF options continue in this issue?
Some comments to those specific options.
Channel / Medium options a. Frequency in Hz as 1x 32-bit float for EPB and IDB b. Channel Spacing in Hz as 1x 32-bit float for IDB or EPB c. Channel Bandwidth in Hz as 1x 32-bit float for IDB or EPB d. Channel encoding as string for IDB or EPB e. Channel name as string for IDB or EPB
a. 32-bit is insufficient - even as an unsigned integer it wouldn't represent frequency above ~4.295GHz (i.e. 5GHz band), perhaps an option in CIB/IDB for the multiplier factor for frequency values? b. are there sub-Hz channel spacing implementations? Is 32-bit integer sufficient? c. is 32-bit integer sufficient? f. Data rate in bps or kbps as 32-bit integer for IDB/CIB, is a multiplier needed here as well?
Capture / Measurement options a. RSSI in dBm as 1x 32-bit antenna index and 1x 32-bit float for EPB b. Channel index as 1x 16-bit uint for EPB
a. separate antennas could also be handled as separate interfaces but perhaps not for MIMO configuration? So multiple instances of the RSSI option could appear in each EPB.
RSSI could be 2x 16-bit to save a 4 bytes, but, 32-bit floats are probably easier to deal with in most situations (unless you're missing an FPU).
I agree with the first part... Is RSSI as float needed? Do any chipsets report RSSI in values other than integers? Most situations not involving low powered embedded network devices without an FPU. I would prefer to see values represented as integers where possible.
Also add for TDMA and TSCH channel access methods.
- slot number as 64-bit integer
- start of slot timestamp (separate from start of frame - which is packet timestamp) - same format an resolution as EPB timestamp (high & low).
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pcapng/pcapng/issues/48#issuecomment-449113098, or mute the thread https://github.com/notifications/unsubscribe-auth/AVScY_a6DglHyJQWS-gLUfSyQ0duZGSkks5u6-eZgaJpZM4QbL0b .
guyharris
commented
5 years ago Right now, if I recall, radiotap handles this by appending additional capabilities blocks w/ only the antenna signal filled out;
Yes.
generating multiple records per packet in pcapng wouldn't be a problem.
Presumably, by analogy to what radiotap does, you mean "generating multiple options per packet block in pcapng wouldn't be a problem".
kismetwireless
commented
5 years ago Yep; sorry for being imprecise. Appending an option block per antenna should be fine.
Having an antenna number may be useful in each block, rather than iterating them; I don't know of any cards currently which report non-sequential antennas but there's always surprises.
On Thu, Dec 20, 2018 at 3:09 PM Guy Harris notifications@github.com wrote:
Right now, if I recall, radiotap handles this by appending additional capabilities blocks w/ only the antenna signal filled out;
Yes.
generating multiple records per packet in pcapng wouldn't be a problem.
Presumably, by analogy to what radiotap does, you mean "generating multiple options per packet block in pcapng wouldn't be a problem".
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pcapng/pcapng/issues/48#issuecomment-449121096, or mute the thread https://github.com/notifications/unsubscribe-auth/AVScY9gaVDB9p-N-wdUQSVuzIwBPWjaqks5u6-6VgaJpZM4QbL0b .
jkcko
commented
5 years ago I agree.. multiple rssi (antenna#/dBm) blocks rather than "list" of dBm with number of antennas derived from the option length.
Additionally, an End of Frame Timestamp (EOF) for time-slotted access methods is desirable since a packet transmission may overflow the timeslot length.
ryankurte
commented
5 years ago On your first set of comments, for all of those I suggested using an IEEE standard 32-bit float, not a 32-bit integer, which should comfortably store all the values we might require.
I think given their prevalence even on embedded systems nowdays it makes sense to use an floats rather than come up with our value / exponent encoding. Should you not have an FPU many of these values could be pre-computed at startup or compile time.
For RSSI, if you're playing with localisation there's a big difference between -10dBm and -10.5dBm, and simulations can have arbitrary accuracy, so again I would personally rather see floats.
I agree.. multiple rssi (antenna#/dBm) blocks rather than "list" of dBm with number of antennas derived from the option length.
Seems like a good approach ^_^ also means you can elide antennas if the packet wasn't seen on them.
Update:
This has been split into two components, per-interface information is approached by introducing a Capture Information Block (CIB) attached to a given interface in #51, per-packet information through the addition of new options to the existing Extended Packet Block (EPB) in #56
The following is left for reference
Original:
Hi there,
I would like to propose the addition of a set of option fields useful for wireless packet capture. A number of these are available in the 802.11 radiotap (meta)header, however are generally useful outside the 802.11 protocol.
There is an argument here that "there's already code to handle that in programs that read pcap and pcap-ng files, there's already code in drivers to generate that information when capturing packets" which doesn't really stand for non-802.11 protocols. By supporting these options as part of the core pcap-ng protocol we would enable development of interesting tools for visualisation and analysis of wireless networks that are protocol independent.
The option fields I would propose initially are:
Supporting GPS Location and frequency in the IDB or EPB allows for specification of multiple static capture points or moving captures at different frequencies.
Cheers,
Ryan