Synchronized Tx/Rx of multiple bladeRFs for MIMO using Mini Expansion Header

[polygon]

I've implemented functionality that allows for sample-accurate simultaneous start of transmission/reception of multiple bladeRF units. To do that, they are chained together on one of the expansion pins (using the Mini-Expansion header right now). After initialization of Tx/Rx, the "master" then pulls the pin high which triggers all the other blades. This allows for sample-accurate start of transmission and reception and can be useful to do first steps in wireless comm. without worrying about synchronization too much. Another application is MIMO where all antennas need to be synchronized as well.

Internally this works by artificially holding the tx_sample_fifo.rempty line high for the transmitter and by keeping the lms_rx_enable line low for the receiver on the FPGA until the trigger arrives. It starts up inactive and hence is fully compatible to the existing firmware / libraries.

The code is basically finished on FPGA and libbladeRF. It has been tested and it shows a constant zero-sample delay when synchronizing only Transmitters or Receivers. If mixed systems are triggered, there is a constant delay of 12 I/Q-Samples (at 20MHz samplerate) between Tx and Rx which is probably caused by the samples propagating from the TxFIFO to the lms6002d.

Is there interest to incorporate this into the bladeRF firmware? I'd be happy to provide the code. Just tell me what works best for you (pull-req, own branch, ...)

[jynik]

There's most certainly interest in pulling this into mainline!

I think the best course of action would be:

• Fork the current repo
• Add your changes to a branch in that fork (with the parent being the up-to-date master)
  • Feel free to add yourself to the CONTRIBUTORS file
  • If possible, try to stage this branch with a reasonably small number of well-organized commits, with detailed commit messages.
• Post a link here for some review and discussion, if needed
• If you haven't already, review and submit a signed Individual CCA via email to bladeRF@nuand.com. Feel free to email with any questions or concerns regarding that.
• From there, we'll probably pull changes into a development branch on the Nuand repo, and schedule the integration of those changes depending on the state of some other tasks.

Thanks!

[polygon]

I uploaded the code to https://github.com/polygon/bladeRF/tree/dev-synctrx which is rebased onto the current master. It currently lacks documentation and examples. I left the commits as they developed internally, if you'd like less commits or a change in structure, I can do that.

[michelp]

Hi @polygon, I'm interested in something very similar to your PR, but instead of a master board triggering a slave, I'd like the trigger to be a 1 pps signal from a gps. I'm not a vhdl expert but it seems to me your code either already does or could easily support this form of triggering. Do you have any thoughts on that?

+1 on this functionality overall! I'm using your patch for further study of what I need.

[jynik]

@polygon - Just wanted to bump this, just to see if you, @robertghilduta, and @bpadalino have had a chance to discuss this further.

[DrizztVD]

@polygon - Could you pls expand a bit on your explanation on how the code works? I'd like to use it but I don't know which pins exactly you are using?

[jynik]

I'm dropping this from the 2014.10 (more likely to become a 2014.11) milestone due to lack of inactivity from the folks involved here.

@bpadalino, where do we stand on this? Were you waiting on a ICAA or ECAA from @polygon?

[staticfloat]

I have a team here that is starting to do some MIMO experiments, and this functionality would be really useful for us to have. What does this PR need in order to be a candidate for merging?

[bpadalino]

We never received an official CAA and the code has kind of gotten slightly stale.

I don't know the legality of taking this (abandoned?) code, reimplementing it, and signing the CAA.

Either that or have the code licensed in a compatible way - such as MIT.

Does that make sense?

[staticfloat]

Yes. I'll wait around and see if @polygon either refreshes this PR, or simply releases all rights to his code to me or you to submit to the project in his stead. If neither happens, I may end up trying to recreate his changes independently, but that would be a drastic action for me as I'm not very fluent in HDL-type things.

[DrizztVD]

I tried to recreate it, holding the LMS enable pin did not seem to work. The code implementation is not well described, it requires much digging around in the HDL to figure out how the pins are supposed to be used. In the end I could not determine exactly at which point the LMS is supposed to be triggered given that a unique NIOS implementation is used into which I could not trace the signal. But then again, the GitHub code that was uploaded by Polygon did not seem to work as is either, one of the boards needs to be set as the master board to provide a time alignment pulse to the second board - how to set this is not described. It's certainly not the MIMO clk master instruction. It's possible that polygons's libbladeRF modifications are rather important in getting the code to work as well.

The best I could do was align counters on the FPGAs of two boards to within about 6 us. The biggest problem turns out to be the USB interface, even if the data streams are aligned on the boards the USB output can still be about 200 ms out of alignment. At this point such synchronization is only possible on platforms that share the same clock and outputs the ADC data to the same FPGA. The only long term solution I see is a ground-up redesign of the entire processing chain where a shared pps signal ensures clock-for-clock time alignment. Any other solutions seem to be temperamental.

On 18 November 2014 00:46, Elliot Saba notifications@github.com wrote:

Yes. I'll wait around and see if @polygon https://github.com/polygon either refreshes this PR, or simply releases all rights to his code to me or you to submit to the project in his stead. If neither happens, I may end up trying to recreate his changes independently, but that would be a drastic action for me as I'm not very fluent in HDL-type things.

— Reply to this email directly or view it on GitHub https://github.com/Nuand/bladeRF/issues/221#issuecomment-63391031.

This message and attachments are subject to a disclaimer. Please refer to http://www.it.up.ac.za/documentation/governance/disclaimer/ for full details.

[bpadalino]

@polygon mentions in the original note that he uses the mini expansion header (J71 on the schematic) to send the synchronization pulses between multiple boards.

For timestamped operation, this signal can be used as a gate or hold off or reset - however you want to view it - to fan out to the other multiple boards.

Since all the boards should be using the same clockbase and time reference from the master/slave MIMO commands, there should be no sample alignment slipping between the multiple units, though the actual sampling time may be +/- a sample or two depending on the register chain synchronizing the asynchronous input from the mini expansion header.

Does that solution make sense?

[DrizztVD]

Hi. In theory it does yes, in practice that did not work for me. I found the J71 pin in the code when I was testing it but the setup information to use the code is insufficient. How is it used to perform a gate, hold off or reset as you mention for instance? I actually did implement my own code with a trigger on the lms enable, the code managed to time out a receive operation if the lms is disabled for more than 1 second or so - but thats about it. The smaller sample misalignment shouldn't be an issue if it can be detected and compensated, but in some cases this requires a lot of additional code to do. The biggest issue is with how the USB delay is accounted for?

On 21 November 2014 at 16:40, Brian Padalino notifications@github.com wrote:

@polygon https://github.com/polygon mentions in the original note that he uses the mini expansion header (J71 on the schematic) to send the synchronization pulses between multiple boards.

For timestamped operation, this signal can be used as a gate or hold off or reset - however you want to view it - to fan out to the other multiple boards.

Since all the boards should be using the same clockbase and time reference from the master/slave MIMO commands, there should be no sample alignment slipping between the multiple units, though the actual sampling time may be +/- a sample or two depending on the register chain synchronizing the asynchronous input from the mini expansion header.

Does that solution make sense?

— Reply to this email directly or view it on GitHub https://github.com/Nuand/bladeRF/issues/221#issuecomment-63978241.

This message and attachments are subject to a disclaimer. Please refer to http://www.it.up.ac.za/documentation/governance/disclaimer/ for full details.

[bpadalino]

The LMS isn't the thing you need to keep held off. It's the incrementing of the counter in the FPGA.

If the same timebase is driving the Si5338 of the different boards, the sample clocks will not experience any drift with respect to each other.

The J71 pin can be shared between the FPGAs to signal a trigger or a 'start' pulse or however you want but you want to not start incrementing counters OR accepting samples into the receive FIFO until the J71 pin has been asserted.

The LMS can be happily receiving with the FPGA dropping all the samples (hence the gate-like functionality).

I think the series of events should be to enable both of the RX chains for both boards, but hold off the actual reception of samples into the FPGA using the J71 pin on the slave, or some NIOS GPIO pin on the master. You then tell the master you want to receive samples by toggling that NIOS GPIO pin which should map to the J71 pin. This start the actual reception and buffering of samples along with the counters in the FPGA (as long as it's coded that way). Then just read as you normally would from both of the devices and they should be within a clock cycle of each other.

Does that make sense or am I not appreciating some nuance you're trying to tell me?

Sorry if I am being obtuse.

[polygon]

First of all, sorry for disappearing so suddenly while leaving this unfinished. Got assigned to another project and didn't have any time to finish this properly.

I've recently started looking into this again and am currently working to adapt the code to the latest Git Master. As @bpadalino already said, the code went stale pretty quickly especially on the FPGA side. Well, that's what you get when not looking after your code I guess :-).

Just to address some questions:

I was measuring in a testbed consisting of four BladeRF, two in transmit and two in receive mode. They were all driven from a common clock-source and all synchronized to a single master. Each TX was transmitting a sequence with a single auto-correlation peak (at zero lag) and basically zero cross-correlation between the transmitters (in one of the simplest cases, just a narrowly pulse-shaped QPSK-stream with random data for each transmitter). By cross-correlating the received signals with the known transmitted streams they can be located very precicely in time-domain (this is essentially what GPS does as well). It showed that neither between transmitters nor receivers there was any delay.

However, this won't get rid of a random phase factor between any two radios (since the PLL of the RF frequency synthesizer usually locks in some random phase). And you might see a subsample delay (since the PLL of the sampling clock locks in some random phase). The former can be fixed with a complex multiplication. The latter, in case of the BladeRF, can be addressed in hardware by using the clock glitching feature of the Si5338.

Anyways, as soon as I've got the code ported to the current Git Master I'll be sure to update here.

[polygon]

I've finished porting the trigger code to the current upstream head. I spent some thought on how to demonstrate it since I usually use a bigger environment allowing remote control of multiple BladeRF units over the network (e.g. using Matlab/Octave/Python) which I don't want to make a dependency for testing this.

In the end I decided to add trigger support to bladeRF-cli and make an ipython notebook that demonstrates triggering.

The current branch for the trigger implementation including bladeRF-cli integration is https://github.com/polygon/bladeRF/tree/port_synctrx. The notebook can be found in https://github.com/polygon/bladeRF/tree/synctrx_doc/doc/synchronized_trx which is also merge-ready if desired. This also contains the transmitted and received waveforms if you want to play with them yourself.

The notebook can also be viewed online here: http://nbviewer.ipython.org/github/polygon/bladeRF/blob/synctrx_doc/doc/synchronized_trx/synctrx_demo.ipynb

[michelp]

This is really amazing stuff @polygon, I've got two bladerfs here i'm going to give it a try this weekend.

Not sure if you know this but I've written some cffi python bindings for libbaderf, I might also take a stab and rewriting your notebook to interface the devices directly from python.

https://github.com/michelp/pybladerf

Thanks again!

[jynik]

@polygon This is fantastic - thank you!

I see you've been in touch with @bpadalino on IRC regarding the ICAA and moving forward. We'll be working to get this merged soon thereafter!

[polygon]

@michelp Great, let me now if there is any issues.

Your library looks slick. I was looking for something like that for my past-time usages. Maybe I'll try it out this weekend if I find some time.

[polygon]

I've rebased the branch onto the new upstream master and changed the trigger polarity to active-low.

The reason is that the Cyclone IV supports internal pull-ups on the I/O pins and if trigger-slaves come up before there is a master setting the bus to a defined value, they would read an undefined line-state (unless you provide external pull-resistors). With the trigger being active-low, in these scenarios the weak pullup on mini_exp1 can be enabled and the trigger will always be well defined whether or not a trigger master is present. No external components required.

I've put the code that actually enables the weak pull-up on mini_exp1 in a separate commit so if that is not generally desired, it can be left out.

If there is already code written using this, it should continue to work. Only if you use the line-state readback in your code, you need to invert polarity (see the changes to bladeRF-cli, it is really trivial). And of course, if you used an external pull-down, that needs to become an (internal or external) pull-up.

https://github.com/polygon/bladeRF/tree/port_synctrx

[jynik]

Pulled these changes into a development branch on the Nuand repo for further review and any changes. In doing so, I squashed and re-arranged a few commits just to tidy up a bit.

Below is a checklist of items that we still need to address:

• General testing of this
• libbladeRF Doxygen warnings:
  • BLADERFTRIGGER* macros need Doxygen comments describing each.
• Discuss bladerf_trigger
  • Was this introduced instead of using bladerf_module to allow for different types of triggers in the future? If not, we could just use bladerf_module here. @polygon, mind commenting on this?
• bladeRF-cli: Regenerate fallback docs from interactive.md
• Review of CLI trigger help text
  • Note min_exp pin assignments?
  • "BladeRF" -> "bladeRF"
• Bulk transfer stalls - perhaps we can introduce an environment variable to define/disable timeouts? I think it's a bit "nicer" to be able to tell people to export an env var, rather than need to recompile with options. (I'm thinking about the case where people are using libbladeRF from a package, pybombs, etc.)
• Need license headers on the trigger HDL items

[ohnishiayano]

@polygon I'm trying to run the notebook and some commands you create. Sorry for asking a basic question, could you tell me how to generate 'pulse.npy' loaded at the beginning of the notebook?

[michelp]

@ohnishiayano it's in the same folder in the repo as the notebook:

https://github.com/polygon/bladeRF/blob/synctrx_doc/doc/synchronized_trx/pulse.npy

[ohnishiayano]

@michelp Thank you for your quick response.

[jjvanthof]

I just tried this yesterday and I'm happy to say it worked perfectly. Now I've got my challenge set up for me; expanding this to a 15 receiver interferometry setup.

[jynik]

Just a heads up to everyone - we'll need to change the Mini Exp pin this utilizes, as we're using pin 1 for the 1PPS/10 MHz VCTCXO tamer.

[ohnishiayano]

When I tried the notebook without any change, received signals were synchronized properly. (Upper diagram) But sometimes results of cross-correlation were shifted by one sample. (Lower diagram) This is just a quick report for your information.

[jynik]

@ohnishiayano, I believe @polygon documented this under in the last heading, Fractional Sampling Offset, section of the IPython notebook doc. Does that clarify this for you?

[ohnishiayano]

@jynik Thank you for letting me know. I missed reading the interpretation. Now I understood this result can happen.

[jynik]

Sorry for the continued delay on this one folks.

We're cleaning up a few HDL items and adding libbladeRF API controls for this.

[jynik]

I'm very sorry for the incredibly huge delay on this folks. This has been merged as of libbladeRF v1.7.0. (2d3f16b47cf9748ef3ab90f6613189a3f053002d).

A simple example with the CLI is included in the host/examples/bladeRF-cli/sync_trx directory.

I am a little uneasy about pulling the IPython notebook into the repo, given that the binary blobs and base64-encoded images will create some git noise for each change. As such this is absent in the recent merge, and I will instead "porting" this great example into a nice little stand-alone document in the nuand-papers repo (attributing @polygon as the primary author, of course).

Again, a big thank goes out to @polygon for his fantastic work on this. Sorry again that this took so long for me to pull in.

[Charlotte5822]

Hello, I'm now working on a multiple bladeRF problem, in which we try to connect 4 bladeRF micro 2.0 to form a 4*4 MIMO. I want to ask about the hardware connection structure of this application. Like whether you use a 1-to-4 divider and the type of divider. Many thanks.