search

jankae / LibreVNA

100kHz to 6GHz 2 port USB based VNA
GNU General Public License v3.0
1.08k stars 204 forks source link

PyVISA-py support for LibreVNA #243

Closed oberstet closed 6 months ago

oberstet commented 6 months ago

Summary

Support for talking to LibreVNA over VISA using PyVISA-py in pure Python without native blobs or libraries.

Paths to implement it

A. If LibreVNA supported running as a USB-TMC (Test and Measurement class) USB device, then it should be possible to talk to it via PyVISA-py USB-TMC from PyVISA.

B. It seems PyVISA-py also support some UDP and/or TCP based backend, and then LibreVNA could expose itself via such UDP/TCP as well, but since LibreVNA does not have a native NIC, the only option on that road would be USB-ETH or similar. However, this still might allow for more powerful scenarios, such as running 5 LibreVNAs connected to local PCs connected in a standard TCP/IP network, and then access all 5 VNAs from 1 (other) PC within the network.

C. Of course the problem of "remote instrument access" in "a network" could also be approached using modern real-time protocols (not raw/plain TCP), and both in LAN and in WAN scenarios. But this is a whole world / different dimension with lots of more aspects of course;)

Extra information

The Virtual Instrument Software Architecture (VISA) is a multivendor, standardized I/O software interface - be it used via native VISA implementation blobs, or via pure Python backends - is focused on "remote in local sense" instrument access, that is directly connected or connected in a LAN - right?

It is not built to access multiple instruments distributed for "remote in WAN sense" scenarios?

Further, it seems (haven't looked deeply though), VISA isn't built for real-time communication with an instrument, right?

I cannot open a web page in my browser on a notebook receiving real-time updates (streaming data) from an instrument located somewhere else on the Internet (in WAN) with VPN-level security, right?

btw / OT: thanks for creating LibreVNA!! I just recently got my instrument, and I am currently "getting started" .. this VNA and hardware stuff is all new to me, I am coming from a software/digital world-only .. OSS and SW dev ..

jankae commented 6 months ago

Thanks for the idea! The LibreVNA-GUI already includes a SCPI server which you can use to control it with (for example) python. Examples on how to use this are available here: https://github.com/jankae/LibreVNA/tree/master/Documentation/UserManual/SCPI_Examples

I think this scenario could be achieved with that:

However, this still might allow for more powerful scenarios, such as running 5 LibreVNAs connected to local PCs connected in a standard TCP/IP network, and then access all 5 VNAs from 1 (other) PC within the network.

Start the GUI on each of these local PCs and connect each to one VNA. Then you can talk to the SCPI servers of all PCs from any other PC in the network.

I don't think anything like this

If LibreVNA supported running as a USB-TMC (Test and Measurement class) USB device

will be very useful. The LibreVNA on its own can not provide S-parameters, it relies on the GUI to perform the necessary calculations (especially for the calibration). And once you have the GUI running, you also have the SCPI server.

oberstet commented 6 months ago

Thank you for the super quick reaction, and for your comments! Some further thoughts .. or questions from my side ...

I should stress: I do have massive experience in SW, networking and distributed protocols, but pretty much zero in RF and embedded HW. but after some months, I now have some antenna for "my awesome new HW". well, the working openEMS model. so I am currently moving to next stage: real PCBs, HW prototyping & measurements, etc. So pls bare with me and do take my comments with a grain of salt;)

So if I understand correctly, what you hint at / the LibreVNA arch looks like:


    ┌───────────────────────────┐                     
    │                           │                     
    │                           │                     
    │                           │                     
    │                           │                     
    │        LibreVNA           │                     
    │                           │      X N            
    │                           │                     
    │                           │                     
    │                           │                     
    │                           │                     
    └────────────┬──────────────┘                     
                 │                                    
                 │                                    
                 │                                    
                 │  USB with ??? profile (serial??)   
                 │                                    
                 │                                    
                 │                                    
                 │                                    
    ┌────────────▼─────────────┐                      
    │                          │                      
    │                          │                      
    │    LibreVNA GUI with     │                      
    │    embedded SCPI server  │                      
    │                          │                      
    │                          │     X N              
    │         PC/OS with       │                      
    │ window manager (X wdws)  │                      
    │                          │                      
    │                          │                      
    │ TCP listening IP & port  │                      
    └────────────▲─────────────┘                      
                 │                                    
                 │                                    
                 │                                    
                 │  SCPI over TCP                     
                 │                                    
                 │                                    
                 │                                    
                 │                                    
                 │                                    
    ┌────────────┴─────────────┐                      
    │                          │                      
    │                          │                      
    │                          │                      
    │                          │                      
    │       Client app         │                      
    │                          │                      
    │                          │                      
    │                          │                      
    │                          │                      
    └──────────────────────────┘

Is that right?

Can you enlighten me rgd "USB with ??? profile (serial??) " ?

jankae commented 6 months ago

Yes, that looks correct. The USB is a custom interface, which is documented here: https://github.com/jankae/LibreVNA/blob/master/Documentation/DeveloperInfo/Device_protocol_v13.pdf Please ignore the sections regarding Ethernet in there, they are only for planned future prototypes or closed source projects. There is no need to implement that USB protocol yourself, that is what the GUI is for.

Instead of using one PC per VNA you could also connect multiple VNAs to one PC and run multiple instances of the GUI (one for each VNA). Or, for a setup with a small footprint, use Raspberry PIs to run the GUI and embedded SCPI server and talk to them over the network.

oberstet commented 6 months ago

Fantastic! Thanks a lot. I love the LibreVNA docs. Including the proper formatting, aka TeX'ed =)

Please bare with me for one more round of Qs / comments;)

Prefix: I think I now understand the LibreVNA tech arch (in this context/at this level), and fwiw, it makes a lot of sense, and is welcome and should be kept supported. Thank you!

Rgd the USB interface (and yeah, I wouldn't be keen on fiddling at the "custom USB device level" .. but) .. just to complete my understanding:

4.1 SweepSettings: Transmitting this packet will switch the LibreVNA into VNA mode and start the sweep. During the sweep, VNADatapoint packets are generated for each completed point in the sweep.

4.13 SpectrumAnalyzerResult : This packet is transmitted by the LibreVNA for every point in the sweep when in spectrum analyzer mode.

Does that mean:

  1. "SpectrumAnalyzerResult" packet is the same as "VNADatapoint" packet?
  2. "VNA mode" is the same as "spectrum analyzer mode"?
  3. When in VNA/spectrum analyzer mode, sweeps will occur automatically over and over again, and within each sweep, SpectrumAnalyzerResult packets will be generated all the time and could be received from LibreVNA via USB (eg only using PyUSB) via custom USB device class packets?

While I should concentrate on "my antenna thing", I can't help but commenting, also now that I started this thread, anyways:

as far as I see, this would be possible using WebSocket and pure Python with PyUSB (and only that). just saying. I should concentrate on getting my antenna working (note to self) ;)

jankae commented 6 months ago

"SpectrumAnalyzerResult" packet is the same as "VNADatapoint" packet? "VNA mode" is the same as "spectrum analyzer mode"?

No, both modes work completely different. You can probably find more detailed explanations online but in short these are the main differences between a VNA and a spectrum analyzer:

VNA:

Spectrum Analyzer:

The LibreVNA is primarily designed as a VNA. I have added the spectrum analyzer mode because the required hardware for that is reasonable close to that of a VNA and it might be useful for quick measurements. However, the performance in terms of accuracy, phase noise and sweep speed is nowhere close to a dedicated spectrum analyzer.

In terms of USB packets:

The SpectrumAnalyzerResult packet is reasonable simple, it just contains the signal levels at the ports and some additional information (such as the measurement frequency). Parsing a VNADatapoint is much more complicated due to its flexible nature. It does not contain S-parameters directly. Instead, the raw (complex) receiver values are transmitted. It is the job of the GUI to calculate the S-parameters from that and also apply additional corrections (such as the calibration without which no meaningful VNA measurements are possible).

When in VNA/spectrum analyzer mode, sweeps will occur automatically over and over again, and within each sweep, SpectrumAnalyzerResult packets will be generated all the time and could be received from LibreVNA via USB (eg only using PyUSB) via custom USB device class packets?

Yes, the sweep will continue (and repeat) until either a different mode is selected or a SetIdle packet is received. The type of packet generated during the sweep depends on the active mode, see above.

  • being able to access LibreVNAs remotely (in a web/WAN scenario) and in real-time streaming fashion would rock
  • I don't really care about VISA nor SCPI

If you drop the real-time streaming requirement, SCPI would be the way to go here. You can't see the sweep come in live but you can get the sweep data directly after it is complete. In general, I would always recommend SCPI for automated measurements. If you just want to take manual measurements and see the data, it is probably not the best solution.

  • I do care about security and networks a lot

I guess this is something which isn't covered in the LibreVNA. There is no encryption or password for the SCPI server and so far that is the only way of accessing it remotely. In my experience, encryption is not a standard feature for lab equipment and most or probably on their own local network anyway.

  • ideally, I could open a web page in a standard web browser on my smartphone, and then receive real-time spectrum analyzer results from Nx LibreVNA devices running "somewhere" (WAN distributed)

Would something like a VNC server work for that purpose? By "receive" do you mean visually show the data or just getting the S-parameter data as e.g. a Touchstone file?

  • use networked "PCs" with connected LibreVNAs where the "PCs" (eg Raspis) do a) not run X (in fact, run "lights out" .. without a human in front) and b) do not expose any listening IP/port (they only run originating/outgoing TCP)

a) you need to run X because the GUI is just one program (i.e. there is way of running only the USB and data handling part without the GUI). You can however configure it in such a way that no further user action is required once it is started. So you could run it on a Raspi (with X but without an actual screen), start it through SSH and then control it with SCPI b) if they don't expose any open port, how would they know where to send the TCP data to?

as far as I see, this would be possible using WebSocket and pure Python with PyUSB (and only that). just saying.

I really don't recommend talking to the LibreVNA by skipping the GUI and using USB directly in any way. It is possible but you would need to replicate a lot of the GUI functionalities. The only thing you are getting over USB are the receiver values. No S-parameters, no calibration, no markers, no advanced math (such as TDR).

oberstet commented 6 months ago

Thank you so much for your explanations and comments! This really helps me putting the pieces together=)

I will answer in detail, please excuse, fwiw, I think I will manage to get my antenna stuff and measurements somehow done using LibreVNA, I hope so!

IOW: consider my lengthy answer not as a concrete obstacle or problem with LibreVNA - at this point where I still haven't done anything using a real PCB yet (which I am working on right now). I will be back;)

Parsing a VNADatapoint is much more complicated due to its flexible nature. It does not contain S-parameters directly. Instead, the raw (complex) receiver values are transmitted. It is the job of the GUI to calculate the S-parameters from that and also apply additional corrections (such as the calibration without which no meaningful VNA measurements are possible).

I see. It would be great if the docs would cover not only SpectrumAnalyzerResult, but VNADatapoint as well, even though it is more complex!

My fallback plan is to use Touchstone data which I can write and read from Python easily.

Thing is: all my code for my openEMS antenna model is using Python and Numpy, eg. have a look at this, which shows how I process Numpy 128-bit complex floating-point numbers arrays with raw U and I values coming from openEMS:

grafik

Down the line, I process and plot that into charts like (this is for a simple reference antenna, not my actual one, which is a wideband LPDA):

grafik

Ultimately, I want to verify that "real-world ~= sim-world" when using the PCB stack XXX from manufacturer YYY.

It likely won't be initially (my expectation), because I need to calibrate for the real-world PCB stack (which I modeled in details .. but).

Therefor I want to first do measurements of the following parameters (or characteristics) of different kinds of transmission lines (such as "Microstrip (MS)", "Coplanar Waveguide with Ground (GCPW)" or "Double-sided Parallel Stripline (DSPSL)"):

  1. velocity factor
  2. characteristic impedance
  3. insertion loss

I need measurement results from LibreVNA in Numpy arrays, and then I can process and plot differences as I like. OR:

If you drop the real-time streaming requirement, SCPI would be the way to go here.

Can I export all information to Touchstone files (manually, from the LibreVNA GUI)?

The whole "real-time spectrum" and "WAN distributed/remote scenario" is just sth I find natural to desire/ask for .. but it is not sth I really need now ... I only have 1 LibreVNA;) But I will comment down below now that I started the discussion.

In my experience, encryption is not a standard feature for lab equipment

My view: all data, whatever, where ever, whenever, should always be encrypted (and authenticated) both in-rest and in-transit.

More so: the encryption keys must only be known to the data owner (me or the party I am communicating with), no one else.

I am fine disclosing that I am doing some communication (thus, stealth or anonymity isn't an absolute requirement, though preferred).

their own local network

From a security perspective, "own" is a desirable goal, and "local" is relative (a chimera).

In the old world, networks were built using a "perimeter security" approach (aka firewalls), nowerdays "zero trust" is the best-practice. Your firewall vendor, your network administrator, some rogue employee, etc are all just different kinds of "attacker". How do you defend?

It doesn't stop there, e.g. you "own" your smartphone, and the SIM card inside is local, but "owned" (== fully remotely controlled) by the mobile carrier, and the SIM card sits on the system bus (!). Do you "own" the BIOS in your PC? Do you "own" the CPU microcode in your PC? The answer is likely no. Someone else is controlling that. Some admin, some vendor. And so on. Not you - that's the point.

Would something like a VNC server work for that purpose? By "receive" do you mean visually show the data or just getting the S-parameter data as e.g. a Touchstone file?

VNC can be tunneled over WebSocket (eg view VNC in web browser), I've done that, but VNC sucks;)

With "receive" I meant: receive all information in VNADatapoint in real-time (within 20-100ms after it was produced by LibreVNA) in a Web browser that runs on the other end of the world.

a) you need to run X because the GUI is just one program

That's a problem. Did you consider also offering parts of the UI as a headless shared library which could be used from Python e.g.?

if they don't expose any open port, how would they know where to send the TCP data to?

Yeah, some "tricks" are needed .. that's the beauty of modern networking: it decouples the application layer from the network layer.

Do you know MQTT or WAMP?

I really don't recommend talking to the LibreVNA by skipping the GUI and using USB directly in any way. It is possible but you would need to replicate a lot of the GUI functionalities. The only thing you are getting over USB are the receiver values. No S-parameters, no calibration, no markers, no advanced math (such as TDR).

I see. My main "itch" is: the GUI is a monolith, and the functions you hint at (which are great, and I wouldn't be keen on replicating ..) are great, but not available as a wrapped, separate, native library.

If they would (as a .so with C ABI .. not C++), reuse in basically any language including Python would be easy.

jankae commented 6 months ago

It would be great if the docs would cover not only SpectrumAnalyzerResult, but VNADatapoint as well, even though it is more complex!

They do already, check section 4.26 of the device protocol v13. But again: as long as you are not talking to the LibreVNA directly over USB, you don't really need to know or understand this information.

I need measurement results from LibreVNA in Numpy arrays, and then I can process and plot differences as I like.

See this example on how to read trace data with python (of course by going through the GUI). It results in a list, not a numpy array but I believe I can leave that conversion to you ;)

Can I export all information to Touchstone files (manually, from the LibreVNA GUI)?

Yes

My view: all data, whatever, where ever, whenever, should always be encrypted (and authenticated) both in-rest and in-transit.

That would be the best case but it also adds a lot of complexity. I am sure they are out there but I have yet to see a test equipment device (meant for bench-use as a developer) that offers encryption. Adding encryption e.g. in the USB data stream didn't even occur to me as an option. It certainly is possible but I don't really see the use case.

I see. My main "itch" is: the GUI is a monolith, and the functions you hint at (which are great, and I wouldn't be keen on replicating ..) are great, but not available as a wrapped, separate, native library.

Yes, you are absolutely right here. If I would start the development of the GUI now, I would absolutely split this in some hardware abstraction layer which handles talking to the device and doing all the calculations and a GUI layer that just displays the data.

As it is, these two parts are deeply intertwined. It would take a massive effort to separate them and it would likely introduce a ton of new bugs as well. At this point I can just recommend you to run the GUI anyway and accept the overhead it introduces. There is the command line option --no-gui to prevent any window from showing up if you just want to control it with a script (but the actual GUI code is still running in the background and as far as I know you still need to run X)

oberstet commented 6 months ago

They do already, check section 4.26 of the device protocol v13.

uups, right, sorry I overlooked that! this looks perfect indeed. yeah, "low level", and I understand users should use the "high level" UI. I mean it's not only processing U/I into S11, but also calibration as you note.

This calculation has to be offloaded to the host because the reference and port receiver measurements may be split across different devices when synchronizing multiple LibreVNAs.

Now you've got me hooked again;) I wasn't aware at all that this is a goal or even possible with multiple LibreVNAs. Which sounds very interesting (I really have to focus on "my antenna" for which I do not need such fancy things), but then I'm wondering (from reading the docs. now hopefully closely enough;): wouldn't that need some kind of synchronized timestamp in VNADatapoint?

Or is the sychronization implicit in "Number of this point in the sweep" and hence it is synchronized to the single point-in-sweep across multiple LibreVNAs?

Does that require some 10MHz time sync signal physically connected to all devices?

Adding encryption e.g. in the USB data stream

I only meant encryption starting from PC - but end-to-end encrypted across WANs.

Encryption starting right after the ADC raw processing, over USB, well sure .. against local attackers that would be necessary ... but that is going very far of course.

At this point I can just recommend you to run the GUI anyway and accept the overhead it introduces.

Alright. Understood!

jankae commented 6 months ago

I wasn't aware at all that this is a goal or even possible with multiple LibreVNAs

It is possible but not really that useful (unless you have a very special measurement that could benefit from the additional isolation above 3 GHz due to the increased physical spacing between the ports).

wouldn't that need some kind of synchronized timestamp in VNADatapoint?

Even more important, it needs synchronization for the different stimulus phases (to make sure that the stimulus is active at one port after the other). This is done over USB and by going through the GUI. The relevant packets are SetTrigger and ClearTrigger.

Does that require some 10MHz time sync signal physically connected to all devices?

Yes. Unless you are measuring very low frequencies, in that case the frequency accuracy of the internal frequency reference is good enough.

A perfect measurement would also need a shared LO signal between all device. As this is not possible with the LibreVNA your through measurements between two synchronized devices are scalar only (no phase measurement).

oberstet commented 6 months ago

would also need a shared LO signal between all device

wow. right. I understand. fwiw, I wouldn't have the money to buy 1k LibreVNAs anyways to build my own SKA =) and actually I don't even know if they have all their LOs in phase sync. anyways, that is waaay beyond my playing field ..