srsran / srsRAN_Project

Open source O-RAN 5G CU/DU solution from Software Radio Systems (SRS) https://docs.srsran.com/projects/project
https://www.srsran.com
GNU Affero General Public License v3.0
514 stars 173 forks source link

Questions about using gpsdo #548

Closed baolindong closed 4 months ago

baolindong commented 6 months ago

Issue Description

[Describe the issue in detail] When using the GPS signal as a synchronization signal, I found that the judgment of the sensor reading in the code may be wrong. The specific problem is that the sensor can actually be read successfully and stored, but the code judgment result is that the sensor reading is unsuccessful.

Lower PHY in quad executor mode.

--== srsRAN gNB (commit 374200dee) ==--

Connecting to AMF on 192.168.1.20:38412
Available radio types: uhd.
[INFO] [UHD] linux; GNU C++ version 10.5.0; Boost_107400; UHD_4.4.0.HEAD-0-g5fac246b
[INFO] [LOGGING] Fastpath logging disabled at runtime.
Making USRP object with args 'type=b200,num_recv_frames=64,num_send_frames=64'
[INFO] [B200] Detected Device: B210
[INFO] [B200] Operating over USB 3.
[INFO] [B200] Detecting internal GPSDO.... 
[INFO] [GPS] Found an internal GPSDO: LC_XO, Firmware Rev 0.929b
[INFO] [B200] Initialize CODEC control...
[INFO] [B200] Initialize Radio control...
[INFO] [B200] Performing register loopback test... 
[INFO] [B200] Register loopback test passed
[INFO] [B200] Performing register loopback test... 
[INFO] [B200] Register loopback test passed
[INFO] [B200] Setting master clock rate selection to 'automatic'.
[INFO] [B200] Asking for clock rate 16.000000 MHz... 
[INFO] [B200] Actually got clock rate 16.000000 MHz.
[INFO] [MULTI_USRP] Setting master clock rate selection to 'manual'.
[INFO] [B200] Asking for clock rate 23.040000 MHz... 
[INFO] [B200] Actually got clock rate 23.040000 MHz.
Error: failed to read sensors.
external clock is ready!
Cell pci=1, bw=20 MHz, dl_arfcn=647000 (n78), dl_freq=3705.0 MHz, dl_ssb_arfcn=646944, ul_freq=3705.0 MHz

==== gNodeB started ===
Type <t> to view trace
Late: 0; Underflow: 2; Overflow: 0;
Late: 0; Underflow: 2; Overflow: 0;
Late: 0; Underflow: 1; Overflow: 0;
Late: 0; Underflow: 1; Overflow: 0;

I tried to modify the code. USRP can be successfully set to GPS time, but after setting it to GPS time, there will be a serious delay problem.

Lower PHY in quad executor mode.

--== srsRAN gNB (commit 374200dee) ==--

Connecting to AMF on 192.168.1.20:38412
Available radio types: uhd.
Making USRP object with args 'type=b200,num_recv_frames=64,num_send_frames=64'
[INFO] [UHD] linux; GNU C++ version 10.5.0; Boost_107400; UHD_4.4.0.HEAD-0-g5fac246b
[INFO] [LOGGING] Fastpath logging disabled at runtime.
[INFO] [B200] Detected Device: B210
[INFO] [B200] Operating over USB 3.
[INFO] [B200] Detecting internal GPSDO.... 
[INFO] [GPS] Found an internal GPSDO: LC_XO, Firmware Rev 0.929b
[INFO] [B200] Initialize CODEC control...
[INFO] [B200] Initialize Radio control...
[INFO] [B200] Performing register loopback test... 
[INFO] [B200] Register loopback test passed
[INFO] [B200] Performing register loopback test... 
[INFO] [B200] Register loopback test passed
[INFO] [B200] Setting master clock rate selection to 'automatic'.
[INFO] [B200] Asking for clock rate 16.000000 MHz... 
[INFO] [B200] Actually got clock rate 16.000000 MHz.
[INFO] [MULTI_USRP] Setting master clock rate selection to 'manual'.
[INFO] [B200] Asking for clock rate 23.040000 MHz... 
[INFO] [B200] Actually got clock rate 23.040000 MHz.
gps_gpgga 
gps_gprmc 
gps_time 
gps_locked 
gps_servo 
ref_locked 
Setting USRP time to 1712542143.0s
[INFO] [MULTI_USRP]     1) catch time transition at pps edge
[INFO] [MULTI_USRP]     2) set times next pps (synchronously)
external clock is ready!
Cell pci=1, bw=20 MHz, dl_arfcn=647000 (n78), dl_freq=3705.0 MHz, dl_ssb_arfcn=646944, ul_freq=3705.0 MHz

==== gNodeB started ===
Type <t> to view trace
Error: exceeded maximum number of timed out transmissions.
Late: 4981; Underflow: 1; Overflow: 0;

Setup Details

[Specify details of the test setup. This would help us reproduce the problem reliably] e.g. Network configuration, Operation System, Hardware, RF front-end, library and driver versions My network configuration profile :

amf:
  addr: 192.168.1.20 #127.0.1.100                                               # The address or hostname of the AMF.
  bind_addr: 192.168.1.23 #127.0.0.1                                            # A local IP that the gNB binds to for traffic from the AMF.

ru_sdr:
  device_driver: uhd                                              # The RF driver name.
  device_args: type=b200,num_recv_frames=64,num_send_frames=64    # Optionally pass arguments to the selected RF driver.
  srate: 23.04 #30.72 #23.04     # RF sample rate might need to be adjusted according to selected bandwidth.
  clock: gpsdo #internal #gpsdo
  sync: gpsdo #internal #gpsdo  
  otw_format: sc12  
  tx_gain: 79 #50                                                     # Transmit gain of the RF might need to adjusted to the given situation.
  rx_gain: 60                                                     # Receive gain of the RF might need to adjusted to the given situation.

cell_cfg:
  dl_arfcn: 647000 #647000 中心频率为3705MHz   #632628 #630000 #368640 #632628                                                # ARFCN of the downlink carrier (center frequency).
  band: 78                                                        # The NR band.
  channel_bandwidth_MHz: 20                                       # Bandwith in MHz. Number of PRBs will be automatically derived.
  common_scs: 30                                                  # Subcarrier spacing in kHz used for data.
  plmn: "00101"                                                   # PLMN broadcasted by the gNB.
  tac: 7                                                          # Tracking area code (needs to match the core configuration).
  pci: 1                                                          # Physical cell ID.

log:
  filename: /tmp/gnb.log                                          # Path of the log file.
  all_level: warning                                                 # Logging level applied to all layers. 

pcap:
  mac_enable: false                                               # Set to true to enable MAC-layer PCAPs.
  mac_filename: /tmp/gnb_mac.pcap                                 # Path where the MAC PCAP is stored.
  ngap_enable: false                                              # Set to true to enable NGAP PCAPs.
  ngap_filename: /tmp/gnb_ngap.pcap                               # Path where the NGAP PCAP is stored.

My Operation System is Ubuntu 22.04.4 LTS, RF front-end is usrp B210, uhd version is 4.4.0

Expected Behavior

[What you expect to happen] I want the base station to work properly when using gps signal as sync signal.

Actual Behaviour

[What happens instead e.g. error message] When the clock is changed to gpsdo in the configuration file, an error of failure to read the sensor will be displayed (the actual sensor has been successfully read). After modifying the judgment conditions, usrp can set the time to gps time, but a serious time out occur.

Steps to reproduce the problem

[Tell us how to reproduce this issue e.g. RF setup, application config files]

Additional Information

[Any additional information, configuration or data that might be necessary to reproduce the issue] I think the code that determines the error in reading the sensor is:

lib\radio\uhd\radio_uhd_impl.cpp :
bool radio_session_uhd_impl::set_time_to_gps_time()
{
  const std::string sensor_name = "gps_time";

  std::vector<std::string> sensors;
  if (device.get_mboard_sensor_names(sensors) != UHD_ERROR_NONE) {
    fmt::print("Error: failed to read sensors. {}\n", device.get_error_message());
    return false;
  }

  // Find sensor name. Error if it is not available.
  if (std::find(sensors.begin(), sensors.end(), sensor_name) == sensors.end()) {
    fmt::print("Error: sensor {} not found.\n", sensor_name);
    return false;
  }

  // Get actual sensor value
  double frac_secs = 0.0;
  if (!device.get_sensor(sensor_name, frac_secs)) {
    fmt::print("Error: not possible to read sensor {}. {}\n", sensor_name, device.get_error_message());
    return false;
  }

  // Get time and set
  fmt::print("Setting USRP time to {}s\n", frac_secs);
  if (device.set_time_unknown_pps(uhd::time_spec_t(frac_secs)) != UHD_ERROR_NONE) {
    fmt::print("Error: failed to set time. {}\n", device.get_error_message());
    return false;
  }

  return true;
}

I tried to make some modifications to the code. After modification, I can set the GPS time, but there will be a time out problem.

bool radio_session_uhd_impl::set_time_to_gps_time()
{
  const std::string sensor_name = "gps_time";

  std::vector<std::string> sensors;
  device.get_mboard_sensor_names(sensors);

  // std::for_each(sensors.begin(),sensors.end(),[](std::string test){std::cout<<test<<" \n";});                    
/*   if (device.get_mboard_sensor_names(sensors) != UHD_ERROR_NONE) {
    //std::for_each(sensors.begin(),sensors.end(),[](std::string test){std::cout<<test<<" \n";});
    fmt::print("Error: failed to read sensors. {}\n", device.get_error_message());
    return false;
  } */

  // Find sensor name. Error if it is not available.
  if (std::find(sensors.begin(), sensors.end(), sensor_name) == sensors.end()) {
    fmt::print("Error: sensor {} not found.\n", sensor_name);
    return false;
  }

  // Get actual sensor value
  double frac_secs = 0.0;
  if (!device.get_sensor(sensor_name, frac_secs)) {
    fmt::print("Error: not possible to read sensor {}. {}\n", sensor_name, device.get_error_message());
    return false;
  }

  // Get time and set
  fmt::print("Setting USRP time to {}s\n", frac_secs);
  device.set_time_unknown_pps(uhd::time_spec_t(frac_secs));
/*   if (device.set_time_unknown_pps(uhd::time_spec_t(frac_secs)) != UHD_ERROR_NONE) {
    fmt::print("Error: failed to set time. {}\n", device.get_error_message());
    return false;
  } */

  return true;
}
xavierarteaga commented 6 months ago

Hi,

Many thanks for your interest. I had a look and there were a couple of mistakes.

Could you try the following patch (edited):

diff --git a/lib/phy/lower/processors/uplink/uplink_processor_impl.cpp b/lib/phy/lower/processors/uplink/uplink_processor_impl.cpp
index 3eb15a14f2..89872af40c 100644
--- a/lib/phy/lower/processors/uplink/uplink_processor_impl.cpp
+++ b/lib/phy/lower/processors/uplink/uplink_processor_impl.cpp
@@ -120,7 +120,7 @@ void lower_phy_uplink_processor_impl::process_symbol_boundary(const baseband_gat
                                                               baseband_gateway_timestamp            timestamp)
 {
   // Calculate the subframe index.
-  unsigned i_sf = timestamp / nof_samples_per_subframe;
+  unsigned i_sf = static_cast<uint64_t>((timestamp / nof_samples_per_subframe) % (NOF_SFNS * NOF_SUBFRAMES_PER_FRAME));

   // Calculate the sample index within the subframe.
   unsigned i_sample_sf = timestamp % nof_samples_per_subframe;
diff --git a/lib/radio/uhd/radio_uhd_impl.cpp b/lib/radio/uhd/radio_uhd_impl.cpp
index d7c0b7dcb6..28b95f6bbb 100644
--- a/lib/radio/uhd/radio_uhd_impl.cpp
+++ b/lib/radio/uhd/radio_uhd_impl.cpp
@@ -259,9 +259,6 @@ radio_session_uhd_impl::radio_session_uhd_impl(const radio_configuration::radio&
     }
   }

-  bool        is_locked = false;
-  std::string sensor_name;
-
   // Set sync source.
   if (!device.set_sync_source(radio_config.clock)) {
     fmt::print("Error: couldn't set sync source: {}\n", device.get_error_message());
@@ -271,21 +268,20 @@ radio_session_uhd_impl::radio_session_uhd_impl(const radio_configuration::radio&
   // Set GPS time if GPSDO is selected.
   if (radio_config.clock.sync == radio_configuration::clock_sources::source::GPSDO) {
     set_time_to_gps_time();
-  }

-  // Select oscillator sensor name.
-  if (radio_config.clock.sync == radio_configuration::clock_sources::source::GPSDO) {
-    sensor_name = "gps_locked";
-  } else {
-    sensor_name = "ref_locked";
+    if (!wait_sensor_locked("gps_locked", true, CLOCK_TIMEOUT_MS)) {
+      // It blocks until sync source is locked.
+      fmt::print("Could not lock reference GPS time source.\n");
+      return;
+    }
   }

   // Wait until external reference / GPS is locked.
   if (radio_config.clock.clock == radio_configuration::clock_sources::source::GPSDO ||
       radio_config.clock.clock == radio_configuration::clock_sources::source::EXTERNAL) {
-    // blocks until clock source is locked
-    if (not wait_sensor_locked(sensor_name, true, CLOCK_TIMEOUT_MS)) {
-      fmt::print("Could not lock reference clock source. Sensor: {}={}.\n", sensor_name, is_locked ? "true" : "false");
+    // It blocks until clock source is locked.
+    if (!wait_sensor_locked("ref_locked", true, CLOCK_TIMEOUT_MS)) {
+      fmt::print("Could not lock reference clock source.\n");
       return;
     }
   }
@@ -314,6 +310,7 @@ radio_session_uhd_impl::radio_session_uhd_impl(const radio_configuration::radio&
       radio_config.clock.sync != radio_configuration::clock_sources::source::GPSDO) {
     device.set_time_unknown_pps(uhd::time_spec_t());
   }

   // Lists of stream descriptions.
   std::vector<radio_uhd_tx_stream::stream_description> tx_stream_description_list;
diff --git a/lib/radio/uhd/radio_uhd_tx_stream.cpp b/lib/radio/uhd/radio_uhd_tx_stream.cpp
index 6326d42f50..d4d9e5ba0c 100644
--- a/lib/radio/uhd/radio_uhd_tx_stream.cpp
+++ b/lib/radio/uhd/radio_uhd_tx_stream.cpp
@@ -81,7 +81,8 @@ void radio_uhd_tx_stream::run_recv_async_msg()

   // Enqueue the task again.
   if (not async_executor.defer([this]() { run_recv_async_msg(); })) {
-    fmt::print(stderr, "Unable to run recv async UHD stream task");
+    fmt::print(stderr, "Unable to run recv async UHD stream task\n");
+    state_fsm.async_task_stopped();
   }
 }

@@ -347,4 +348,4 @@ void radio_uhd_tx_stream::wait_stop()
 unsigned radio_uhd_tx_stream::get_buffer_size() const
 {
   return max_packet_size;
-}
\ No newline at end of file
+}
diff --git a/lib/ru/generic/ru_controller_generic_impl.cpp b/lib/ru/generic/ru_controller_generic_impl.cpp
index 10b9d0b434..5c9d3d6eda 100644
--- a/lib/ru/generic/ru_controller_generic_impl.cpp
+++ b/lib/ru/generic/ru_controller_generic_impl.cpp
@@ -24,11 +24,14 @@ ru_controller_generic_impl::ru_controller_generic_impl(std::vector<lower_phy_con

 void ru_controller_generic_impl::start()
 {
-  // Calculate starting time from the radio current time plus one hundred milliseconds and rounded to the next subframe.
+  // Calculate starting time from the radio current time plus one hundred milliseconds.
   double                     delay_s      = 0.1;
   baseband_gateway_timestamp current_time = radio.read_current_time();
   baseband_gateway_timestamp start_time   = current_time + static_cast<uint64_t>(delay_s * srate_MHz * 1e6);
-  start_time = divide_ceil(start_time, static_cast<uint64_t>(srate_MHz * 1e3)) * static_cast<uint64_t>(srate_MHz * 1e3);
+
+  // Round start time to the next subframe.
+  uint64_t sf_duration = static_cast<uint64_t>(srate_MHz * 1e3);
+  start_time           = divide_ceil(start_time, sf_duration) * sf_duration;

   radio.start(start_time);

Let us know how it goes.

Kind regards, Xavier

baolindong commented 6 months ago

Hi, @xavierarteaga Thank you very much for your help, I've tried the patch you gave me,It solves the problem of incorrect judgment of sensor reading status, but there is still a serious time out problem after usrp uses gps time.

The terminal log:

Lower PHY in quad executor mode.

--== srsRAN gNB (commit 374200dee) ==--

Connecting to AMF on 192.168.1.20:38412
Available radio types: uhd.
[INFO] [UHD] linux; GNU C++ version 10.5.0; Boost_107400; UHD_4.4.0.HEAD-0-g5fac246b
[INFO] [LOGGING] Fastpath logging disabled at runtime.
Making USRP object with args 'type=b200,num_recv_frames=64,num_send_frames=64'
[INFO] [B200] Detected Device: B210
[INFO] [B200] Operating over USB 3.
[INFO] [B200] Detecting internal GPSDO.... 
[INFO] [GPS] Found an internal GPSDO: LC_XO, Firmware Rev 0.929b
[INFO] [B200] Initialize CODEC control...
[INFO] [B200] Initialize Radio control...
[INFO] [B200] Performing register loopback test... 
[INFO] [B200] Register loopback test passed
[INFO] [B200] Performing register loopback test... 
[INFO] [B200] Register loopback test passed
[INFO] [B200] Setting master clock rate selection to 'automatic'.
[INFO] [B200] Asking for clock rate 16.000000 MHz... 
[INFO] [B200] Actually got clock rate 16.000000 MHz.
[INFO] [MULTI_USRP] Setting master clock rate selection to 'manual'.
[INFO] [B200] Asking for clock rate 23.040000 MHz... 
[INFO] [B200] Actually got clock rate 23.040000 MHz.
Setting USRP time to 1712631702.0s
[INFO] [MULTI_USRP]     1) catch time transition at pps edge
[INFO] [MULTI_USRP]     2) set times next pps (synchronously)
Cell pci=1, bw=20 MHz, dl_arfcn=647000 (n78), dl_freq=3705.0 MHz, dl_ssb_arfcn=646944, ul_freq=3705.0 MHz

==== gNodeB started ===
Type <t> to view trace
Error: exceeded maximum number of timed out transmissions.
Late: 4996; Underflow: 0; Overflow: 0;
Error: exceeded maximum number of timed out transmissions.
Late: 4995; Underflow: 1; Overflow: 0;
Error: exceeded maximum number of timed out transmissions.
Late: 5000; Underflow: 0; Overflow: 0;
^CStopping   

Kind regards, baolin

xavierarteaga commented 6 months ago

I reproduced the issue and made a larger patch that is being tested. I am sorry for the multiple iterations. My B200 takes very long to lock.

This is the current patch:

diff --git a/include/srsran/support/math_utils.h b/include/srsran/support/math_utils.h
index b63a4e2ba5..2518159c62 100644
--- a/include/srsran/support/math_utils.h
+++ b/include/srsran/support/math_utils.h
@@ -26,11 +26,16 @@ static constexpr float near_zero = 1e-9;

 /// \brief Performs an integer division rounding up.
 ///
-/// \param[in]  num Numerator.
-/// \param[out] den Denominator.
+/// \tparam     NumType Division numerator integer type.
+/// \tparam     DenType Division denominator integer type.
+/// \param[in]  num     Numerator.
+/// \param[out] den     Denominator.
 /// \return The result of the operation.
-inline constexpr unsigned divide_ceil(unsigned num, unsigned den)
+template <typename NumType, typename DenType>
+inline constexpr auto divide_ceil(NumType num, DenType den)
 {
+  static_assert(std::is_integral<NumType>::value, "The numerator must be an integer.");
+  static_assert(std::is_integral<DenType>::value, "The denominator must be an integer.");
   srsran_sanity_check(den != 0, "Denominator cannot be zero.");
   return (num + (den - 1)) / den;
 }
diff --git a/lib/phy/lower/processors/uplink/uplink_processor_impl.cpp b/lib/phy/lower/processors/uplink/uplink_processor_impl.cpp
index 3eb15a14f2..89872af40c 100644
--- a/lib/phy/lower/processors/uplink/uplink_processor_impl.cpp
+++ b/lib/phy/lower/processors/uplink/uplink_processor_impl.cpp
@@ -120,7 +120,7 @@ void lower_phy_uplink_processor_impl::process_symbol_boundary(const baseband_gat
                                                               baseband_gateway_timestamp            timestamp)
 {
   // Calculate the subframe index.
-  unsigned i_sf = timestamp / nof_samples_per_subframe;
+  unsigned i_sf = static_cast<uint64_t>((timestamp / nof_samples_per_subframe) % (NOF_SFNS * NOF_SUBFRAMES_PER_FRAME));

   // Calculate the sample index within the subframe.
   unsigned i_sample_sf = timestamp % nof_samples_per_subframe;
diff --git a/lib/radio/uhd/radio_uhd_impl.cpp b/lib/radio/uhd/radio_uhd_impl.cpp
index 55101498f1..c8df3c87e8 100644
--- a/lib/radio/uhd/radio_uhd_impl.cpp
+++ b/lib/radio/uhd/radio_uhd_impl.cpp
@@ -21,7 +21,7 @@ bool radio_session_uhd_impl::set_time_to_gps_time()
   const std::string sensor_name = "gps_time";

   std::vector<std::string> sensors;
-  if (device.get_mboard_sensor_names(sensors) != UHD_ERROR_NONE) {
+  if (!device.get_mboard_sensor_names(sensors)) {
     fmt::print("Error: failed to read sensors. {}\n", device.get_error_message());
     return false;
   }
@@ -41,7 +41,7 @@ bool radio_session_uhd_impl::set_time_to_gps_time()

   // Get time and set
   fmt::print("Setting USRP time to {}s\n", frac_secs);
-  if (device.set_time_unknown_pps(uhd::time_spec_t(frac_secs)) != UHD_ERROR_NONE) {
+  if (!device.set_time_unknown_pps(uhd::time_spec_t(frac_secs))) {
     fmt::print("Error: failed to set time. {}\n", device.get_error_message());
     return false;
   }
@@ -51,8 +51,7 @@ bool radio_session_uhd_impl::set_time_to_gps_time()

 bool radio_session_uhd_impl::wait_sensor_locked(const std::string& sensor_name, bool is_mboard, int timeout)
 {
-  bool is_locked = false;
-  auto end_time  = std::chrono::steady_clock::now() + std::chrono::milliseconds(timeout);
+  auto end_time = std::chrono::steady_clock::now() + std::chrono::milliseconds(timeout);

   // Get sensor list
   std::vector<std::string> sensors;
@@ -78,6 +77,7 @@ bool radio_session_uhd_impl::wait_sensor_locked(const std::string& sensor_name,

   do {
     // Get actual sensor value
+    bool is_locked = false;
     if (is_mboard) {
       if (!device.get_sensor(sensor_name, is_locked)) {
         fmt::print("Error: reading mboard sensor {}. {}.\n", sensor_name, device.get_error_message());
@@ -90,11 +90,16 @@ bool radio_session_uhd_impl::wait_sensor_locked(const std::string& sensor_name,
       }
     }

-    // Read value and wait
-    std::this_thread::sleep_for(std::chrono::milliseconds(1));
-  } while (not is_locked and std::chrono::steady_clock::now() < end_time);
+    // Return true if the sensor is locked.
+    if (is_locked) {
+      return true;
+    }
+
+    // Sleep some time before trying it again.
+    std::this_thread::sleep_for(std::chrono::milliseconds(10));
+  } while (std::chrono::steady_clock::now() < end_time);

-  return is_locked;
+  return false;
 }

 bool radio_session_uhd_impl::set_tx_gain_unprotected(unsigned port_idx, double gain_dB)
@@ -254,9 +259,6 @@ radio_session_uhd_impl::radio_session_uhd_impl(const radio_configuration::radio&
     }
   }

-  bool        is_locked = false;
-  std::string sensor_name;
-
   // Set sync source.
   if (!device.set_sync_source(radio_config.clock)) {
     fmt::print("Error: couldn't set sync source: {}\n", device.get_error_message());
@@ -265,22 +267,21 @@ radio_session_uhd_impl::radio_session_uhd_impl(const radio_configuration::radio&

   // Set GPS time if GPSDO is selected.
   if (radio_config.clock.sync == radio_configuration::clock_sources::source::GPSDO) {
-    set_time_to_gps_time();
-  }
+    if (!wait_sensor_locked("gps_locked", true, CLOCK_TIMEOUT_MS)) {
+      // It blocks until sync source is locked.
+      fmt::print("Could not lock reference GPS time source.\n");
+      return;
+    }

-  // Select oscillator sensor name.
-  if (radio_config.clock.sync == radio_configuration::clock_sources::source::GPSDO) {
-    sensor_name = "gps_locked";
-  } else {
-    sensor_name = "ref_locked";
+    set_time_to_gps_time();
   }

   // Wait until external reference / GPS is locked.
-  if (radio_config.clock.sync == radio_configuration::clock_sources::source::GPSDO ||
-      radio_config.clock.sync == radio_configuration::clock_sources::source::EXTERNAL) {
-    // blocks until clock source is locked
-    if (not wait_sensor_locked(sensor_name, true, CLOCK_TIMEOUT_MS)) {
-      fmt::print("Could not lock reference clock source. Sensor: {}={}.\n", sensor_name, is_locked ? "true" : "false");
+  if (radio_config.clock.clock == radio_configuration::clock_sources::source::GPSDO ||
+      radio_config.clock.clock == radio_configuration::clock_sources::source::EXTERNAL) {
+    // It blocks until clock source is locked.
+    if (!wait_sensor_locked("ref_locked", true, CLOCK_TIMEOUT_MS)) {
+      fmt::print("Could not lock reference clock source.\n");
       return;
     }
   }
diff --git a/lib/radio/uhd/radio_uhd_tx_stream.cpp b/lib/radio/uhd/radio_uhd_tx_stream.cpp
index 6326d42f50..d4d9e5ba0c 100644
--- a/lib/radio/uhd/radio_uhd_tx_stream.cpp
+++ b/lib/radio/uhd/radio_uhd_tx_stream.cpp
@@ -81,7 +81,8 @@ void radio_uhd_tx_stream::run_recv_async_msg()

   // Enqueue the task again.
   if (not async_executor.defer([this]() { run_recv_async_msg(); })) {
-    fmt::print(stderr, "Unable to run recv async UHD stream task");
+    fmt::print(stderr, "Unable to run recv async UHD stream task\n");
+    state_fsm.async_task_stopped();
   }
 }

@@ -347,4 +348,4 @@ void radio_uhd_tx_stream::wait_stop()
 unsigned radio_uhd_tx_stream::get_buffer_size() const
 {
   return max_packet_size;
-}
\ No newline at end of file
+}
diff --git a/lib/ru/generic/ru_controller_generic_impl.cpp b/lib/ru/generic/ru_controller_generic_impl.cpp
index 10b9d0b434..5c9d3d6eda 100644
--- a/lib/ru/generic/ru_controller_generic_impl.cpp
+++ b/lib/ru/generic/ru_controller_generic_impl.cpp
@@ -24,11 +24,14 @@ ru_controller_generic_impl::ru_controller_generic_impl(std::vector<lower_phy_con

 void ru_controller_generic_impl::start()
 {
-  // Calculate starting time from the radio current time plus one hundred milliseconds and rounded to the next subframe.
+  // Calculate starting time from the radio current time plus one hundred milliseconds.
   double                     delay_s      = 0.1;
   baseband_gateway_timestamp current_time = radio.read_current_time();
   baseband_gateway_timestamp start_time   = current_time + static_cast<uint64_t>(delay_s * srate_MHz * 1e6);
-  start_time = divide_ceil(start_time, static_cast<uint64_t>(srate_MHz * 1e3)) * static_cast<uint64_t>(srate_MHz * 1e3);
+
+  // Round start time to the next subframe.
+  uint64_t sf_duration = static_cast<uint64_t>(srate_MHz * 1e3);
+  start_time           = divide_ceil(start_time, sf_duration) * sf_duration;

   radio.start(start_time);

This patch will go into the next release.

baolindong commented 6 months ago

Hi, @xavierarteaga Thank you very much for your help. Now the delay problem has been solved. However, after the base station is started, my mobile phone cannot connect to the base station normally (when using gps signal as clock and synchronization signal). After I change the clock and synchronization signal to usrp's internal clock, the phone can connect normally.

There are the error messages from the open5gs core network:

smf log:
04/09 16:52:06.788: [smf] INFO: [Added] Number of SMF-Sessions is now 1 (../src/smf/context.c:3068)
04/09 16:52:06.793: [sbi] WARNING: [UDM] (SCP-discover) NF has already been added [5bf2f45e-f0cd-41ee-9b03-550a9fe4ca5b:1] (../lib/sbi/path.c:216)
04/09 16:52:06.801: [smf] INFO: UE SUPI[imsi-001010000000104] DNN[internet] IPv4[10.45.0.4] IPv6[] (../src/smf/npcf-handler.c:539)
04/09 16:52:06.802: [smf] WARNING: Unknown PCO ID:(0x23) (../src/smf/context.c:3011)
04/09 16:52:06.802: [smf] WARNING: Unknown PCO ID:(0x24) (../src/smf/context.c:3011)
04/09 16:52:06.872: [sbi] WARNING: [UDM] (SCP-discover) NF has already been added [5bf2f45e-f0cd-41ee-9b03-550a9fe4ca5b:2] (../lib/sbi/path.c:216)
04/09 16:52:30.716: [smf] WARNING: [imsi-001010000000104:1] No PolicyAssociationId. Forcibly remove SESSION (../src/smf/nsmf-handler.c:874)
04/09 16:52:30.718: [smf] INFO: Removed Session: UE IMSI:[imsi-001010000000104] DNN:[internet:1] IPv4:[10.45.0.4] IPv6:[] (../src/smf/context.c:1672)
04/09 16:52:30.718: [smf] INFO: [Removed] Number of SMF-Sessions is now 0 (../src/smf/context.c:3076)
04/09 16:52:30.718: [smf] INFO: [Removed] Number of SMF-UEs is now 0 (../src/smf/context.c:1080)
upf.log(The address of my base station host is 192.168.1.23 and the address of my core network host is 192.168.1.20): 
04/09 16:37:59.435: [upf] ERROR: [192.168.1.20] Send Error Indication [TEID:0x4ca7] to [192.168.1.23] (../src/upf/gtp-path.c:414)
04/09 16:37:59.435: [upf] ERROR: [192.168.1.20] Send Error Indication [TEID:0x4ca7] to [192.168.1.23] (../src/upf/gtp-path.c:414)
04/09 16:37:59.435: [upf] ERROR: [192.168.1.20] Send Error Indication [TEID:0x4ca7] to [192.168.1.23] (../src/upf/gtp-path.c:414)
04/09 16:37:59.435: [upf] ERROR: [192.168.1.20] Send Error Indication [TEID:0x4ca7] to [192.168.1.23] (../src/upf/gtp-path.c:414)
04/09 16:37:59.436: [upf] ERROR: [192.168.1.20] Send Error Indication [TEID:0x4ca7] to [192.168.1.23] (../src/upf/gtp-path.c:414)
04/09 16:37:59.436: [upf] ERROR: [192.168.1.20] Send Error Indication [TEID:0x4ca7] to [192.168.1.23] (../src/upf/gtp-path.c:414)
04/09 16:52:06.802: [upf] INFO: [Added] Number of UPF-Sessions is now 1 (../src/upf/context.c:208)
04/09 16:52:06.802: [upf] INFO: UE F-SEID[UP:0x5cd CP:0xb5b] APN[internet] PDN-Type[1] IPv4[10.45.0.4] IPv6[] (../src/upf/context.c:485)
04/09 16:52:06.802: [upf] INFO: UE F-SEID[UP:0x5cd CP:0xb5b] APN[internet] PDN-Type[1] IPv4[10.45.0.4] IPv6[] (../src/upf/context.c:485)
04/09 16:52:30.714: [upf] INFO: [Removed] Number of UPF-sessions is now 0 (../src/upf/context.c:252)
xavierarteaga commented 6 months ago

I understand that when you set ru_sdr --clock=gpsdo --sync=gpsdo the UE does not attach and the core network logs the errors above. If you run the gNb with ru_sdr --clock=internal --sync=internal (after using the GPSDO) the UE attaches and the errors are not logged. Is my understanding correct?

My COTS (Fairphone5) does not attach when I use Leo Bodnar GPSDO as an external reference. I still need to investigate further.

Is your UE cabled to the USRP?

baolindong commented 6 months ago

Yes, your understanding is correct.

My UE(Realme Q5 pro) can search for a signal called "open5gs"(the CN's name), but cannot connect to it. I'll also try to find out what the problem is.

baolindong commented 6 months ago

Hi, @xavierarteaga

I would like to ask if there are any results from your investigation on this issue. I wonder if the calculation method of frame number has changed after switching to GPS signal, causing users to be unable to connect normally.

xavierarteaga commented 6 months ago

I would like to ask if there are any results from your investigation on this issue. I wonder if the calculation method of frame number has changed after switching to GPS signal, causing users to be unable to connect normally.

I have not got deeper as my B200's GPSDO is not locking well. I found that the GPS time causes the overflow of some timestamps in the SFN derivation. You could try a zero time in the next PPS.

baolindong commented 6 months ago

My B200 takes a long time to lock the gps signal(GPS antenna placed outdoors), but once it locks it becomes very stable.

What I understand is that when using GPS time to calculate SFN, due to the overflow problem caused by the large GPS time value, UE cannot connect normally. To solve this problem, if the GPS time used to calculate SFN is set to zero, it may be possible. To solve this problem, is my understanding correct?

I use GPS signals as synchronization signals to achieve frame synchronization between base stations. If I set the GPS time to zero at the next pps, will the frame synchronization between base stations be affected?

xavierarteaga commented 6 months ago

What I understand is that when using GPS time to calculate SFN, due to the overflow problem caused by the large GPS time value, UE cannot connect normally. To solve this problem, if the GPS time used to calculate SFN is set to zero, it may be possible. To solve this problem, is my understanding correct?

The idea of the experiment is to validate our hypothesis. If the UE attaches after setting the timestamp to 0, it could be there is an overflow. Otherwise, the problem is not related to the GPS time.

I use GPS signals as synchronization signals to achieve frame synchronization between base stations. If I set the GPS time to zero at the next pps, will the frame synchronization between base stations be affected?

It would be affected but it is an experiment to discard whether the problem is related (or not) to the timestamp.

For this, I suggest commenting on this section:

  if (!device.get_sensor(sensor_name, frac_secs)) {
    fmt::print("Error: not possible to read sensor {}. {}\n", sensor_name, device.get_error_message());
    return false;
  }
pgawlowicz commented 6 months ago

Hi, maybe something related to your issue: https://github.com/srsran/srsRAN_Project/discussions/554

baolindong commented 6 months ago

Hi,@xavierarteaga

Under your suggestion, I tried an experiment. After setting the time of b200 to zero, UE can access normally.

There are some logs of base station:

--== srsRAN gNB (commit 374200dee) ==--

Connecting to AMF on 192.168.1.20:38412
Available radio types: uhd.
[INFO] [UHD] linux; GNU C++ version 10.5.0; Boost_107400; UHD_4.4.0.HEAD-0-g5fac246b
[INFO] [LOGGING] Fastpath logging disabled at runtime.
Making USRP object with args 'type=b200,num_recv_frames=64,num_send_frames=64'
[INFO] [B200] Detected Device: B210
[INFO] [B200] Operating over USB 3.
[INFO] [B200] Detecting internal GPSDO.... 
[INFO] [GPS] Found an internal GPSDO: LC_XO, Firmware Rev 0.929b
[INFO] [B200] Initialize CODEC control...
[INFO] [B200] Initialize Radio control...
[INFO] [B200] Performing register loopback test... 
[INFO] [B200] Register loopback test passed
[INFO] [B200] Performing register loopback test... 
[INFO] [B200] Register loopback test passed
[INFO] [B200] Setting master clock rate selection to 'automatic'.
[INFO] [B200] Asking for clock rate 16.000000 MHz... 
[INFO] [B200] Actually got clock rate 16.000000 MHz.
[INFO] [MULTI_USRP] Setting master clock rate selection to 'manual'.
[INFO] [B200] Asking for clock rate 23.040000 MHz... 
[INFO] [B200] Actually got clock rate 23.040000 MHz.
Setting USRP time to 0.0s
[INFO] [MULTI_USRP]     1) catch time transition at pps edge
[INFO] [MULTI_USRP]     2) set times next pps (synchronously)
Cell pci=1, bw=20 MHz, dl_arfcn=647000 (n78), dl_freq=3705.0 MHz, dl_ssb_arfcn=646944, ul_freq=3705.0 MHz

==== gNodeB started ===
Type <t> to view trace
Late: 0; Underflow: 1; Overflow: 0;
tyLate: 0; Underflow: 1; Overflow: 0;

Available commands:
    t: start/stop console trace
    q: quit application

Late: 0; Underflow: 2; Overflow: 0;
t
Late: 0; Underflow: 1; Overflow: 0;

           -----------------DL-----------------------|------------------UL--------------------
 pci rnti  cqi  ri  mcs  brate   ok  nok  (%)  dl_bs | pusch  mcs  brate   ok  nok  (%)    bsr
   1 4601   15   1   27    68k   22    0   0%      0 |  24.8   28    39k   10    0   0%      0
   1 4601   15   1   28   117k   25    0   0%      0 |  26.6   28    41k   10    0   0%      0
   1 4601   15   1   28    78k   16    0   0%      0 |  27.5   28    35k    9    0   0%      0
   1 4601   15   1   28    20k   18    0   0%      0 |  26.3   28    41k   10    0   0%      0
   1 4601   15   1   28   5.2k    7    0   0%      0 |  26.5   28    17k    4    0   0%      0
   1 4601   15   1   28    35k   29    0   0%      0 |  27.3   27    33k    9    0   0%      0
   1 4601   15   1   28   469k  188    0   0%      0 |  27.0   28   169k   44    0   0%      0
   1 4601   15   1   28    36M 1185    0   0%  1.01M |  20.8   23   1.4M  105    0   0%  1.04k
   1 4601   15   1   28    45M 1400    0   0%   687k |  19.7   23   1.5M  111    0   0%  1.45k
   1 4601   15   1   28    45M 1400    0   0%   959k |  18.8   22   1.4M  103    0   0%  1.04k
   1 4601   15   1   28    45M 1400    0   0%   929k |  21.1   24   1.2M  101    0   0%  2.01k

           -----------------DL-----------------------|------------------UL--------------------
 pci rnti  cqi  ri  mcs  brate   ok  nok  (%)  dl_bs | pusch  mcs  brate   ok  nok  (%)    bsr
   1 4601   15   1   28    45M 1399    1   0%   772k |  21.3   25   1.2M  104    0   0%  1.45k
   1 4601   15   1   28    45M 1400    0   0%   683k |  24.0   27   1.1M  102    0   0%  1.45k
   1 4601   15   1   28    45M 1400    0   0%   732k |  23.7   27   1.1M  100    0   0%  1.04k
   1 4601   15   1   28    45M 1400    0   0%   654k |  25.2   27   990k  100    0   0%  1.45k
   1 4601   15   1   28    45M 1400    0   0%   501k |  25.0   28   1.0M  100    0   0%  1.04k
   1 4601   15   1   28    22M  774    1   0%      0 |  23.0   26   972k   81    0   0%      0
  1 4601   15   1   28   427k  342    0   0%     11 |  17.1   21   7.5M  380    0   0%   300k
   1 4601   15   1   28   615k  490    0   0%      0 |  16.7   21    13M  600    0   0%   300k
   1 4601   15   1   28   499k  445    0   0%      0 |  16.7   22    13M  600    0   0%   300k
   1 4601   15   1   28   501k  463    0   0%      0 |  16.8   23    14M  598    2   0%   300k
   1 4601   15   1   28   548k  487    0   0%      0 |  16.9   23    15M  600    0   0%   300k

           -----------------DL-----------------------|------------------UL--------------------
 pci rnti  cqi  ri  mcs  brate   ok  nok  (%)  dl_bs | pusch  mcs  brate   ok  nok  (%)    bsr
   1 4601   15   1   28   537k  478    0   0%      0 |  17.0   24    16M  600    0   0%   300k
   1 4601   15   1   28   597k  514    0   0%      0 |  17.0   25    17M  600    0   0%   300k
   1 4601   15   1   28   566k  494    0   0%      0 |  17.0   26    17M  600    0   0%   300k
   1 4601   15   1   28   587k  497    0   0%      0 |  16.9   27    18M  600    0   0%   300k
   1 4601   15   1   28   603k  536    0   0%      0 |  16.9   26    18M  600    0   0%   300k
   1 4601   15   1   28   578k  486   22   4%      0 |  16.2   26    17M  594    6   1%   300k

My idea is that the GPS time is a very large number (1713345128.0s). If the remainder of this number is set to the usrp time (for example, setting the time to 45128.0s), will it not affect the frame synchronization between base stations? solve this problem.

I want to know how to solve this problem to facilitate my next work.

Thank you very much!

xavierarteaga commented 6 months ago

I tried to attach my COTS UE with the following patch on the top to the code we will soon release:

diff --git a/lib/radio/uhd/radio_uhd_impl.cpp b/lib/radio/uhd/radio_uhd_impl.cpp
index c8df3c87e8..2e2930b6fd 100644
--- a/lib/radio/uhd/radio_uhd_impl.cpp
+++ b/lib/radio/uhd/radio_uhd_impl.cpp
@@ -311,6 +311,14 @@ radio_session_uhd_impl::radio_session_uhd_impl(const radio_configuration::radio&
     device.set_time_unknown_pps(uhd::time_spec_t());
   }

+  double full_seconds = 1713345128.0;
+
+  fmt::print("Setting USRP time to {}s\n", full_seconds);
+  device.set_time_unknown_pps(uhd::time_spec_t(full_seconds));
+
   // Lists of stream descriptions.
   std::vector<radio_uhd_tx_stream::stream_description> tx_stream_description_list;
   std::vector<radio_uhd_rx_stream::stream_description> rx_stream_description_list;

The UE has attached to the gNb and the ping went successfully. As far as I know, all the fixes related to this issue are included in the comment above.

My radio configuration is the following:

ru_sdr:
  device_driver: uhd
  device_args: type=b200,master_clock_rate=61.44e6
  tx_gain: 75
  rx_gain: 40
  srate: 61.44
  clock: internal
  sync: internal
  time_alignment_calibration: 0
baolindong commented 6 months ago

Hi, @xavierarteaga

I tried to attach my COTS UE with the following patch on the top to the code we will soon release:

diff --git a/lib/radio/uhd/radio_uhd_impl.cpp b/lib/radio/uhd/radio_uhd_impl.cpp
index c8df3c87e8..2e2930b6fd 100644
--- a/lib/radio/uhd/radio_uhd_impl.cpp
+++ b/lib/radio/uhd/radio_uhd_impl.cpp
@@ -311,6 +311,14 @@ radio_session_uhd_impl::radio_session_uhd_impl(const radio_configuration::radio&
     device.set_time_unknown_pps(uhd::time_spec_t());
   }

+  double full_seconds = 1713345128.0;
+
+  fmt::print("Setting USRP time to {}s\n", full_seconds);
+  device.set_time_unknown_pps(uhd::time_spec_t(full_seconds));
+
   // Lists of stream descriptions.
   std::vector<radio_uhd_tx_stream::stream_description> tx_stream_description_list;
   std::vector<radio_uhd_rx_stream::stream_description> rx_stream_description_list;

The UE has attached to the gNb and the ping went successfully. As far as I know, all the fixes related to this issue are included in the comment above.

My radio configuration is the following:

ru_sdr:
  device_driver: uhd
  device_args: type=b200,master_clock_rate=61.44e6
  tx_gain: 75
  rx_gain: 40
  srate: 61.44
  clock: internal
  sync: internal
  time_alignment_calibration: 0

I tried your patch, but on my device, ue still cannot connect normally. After adding the patch, the usrp time is set to 1713345128.0s. In this case, I use the internal signal as a reference signal and ue cannot connect normally. , when I canceled the patch, the ue connection returned to normal.

Because of this phenomenon, I guessed that when the usrp time is only a smaller value, the system can run normally. When the usrp time is set to a larger value, ue cannot connect normally. Then I did an experiment and verified this conjecture is that when the usrp time is set to a value of one hundred thousand levels, ue can connect normally. When it reaches a value of one million levels, ue cannot connect normally.

I tried to attach my COTS UE with the following patch:

diff --git a/lib/radio/uhd/radio_uhd_impl.cpp b/lib/radio/uhd/radio_uhd_impl.cpp
--- a/lib/radio/uhd/radio_uhd_impl.cpp
+++ b/lib/radio/uhd/radio_uhd_impl.cpp

 @@ radio_session_uhd_impl::radio_session_uhd_impl(const radio_configuration::radio&
     device.set_time_unknown_pps(uhd::time_spec_t());
   }

-  double full_seconds = 1713345128.0;
-
-  fmt::print("Setting USRP time to {}s\n", full_seconds);
-  device.set_time_unknown_pps(uhd::time_spec_t(full_seconds));
-
   // Lists of stream descriptions.
   std::vector<radio_uhd_tx_stream::stream_description> tx_stream_description_list;
   std::vector<radio_uhd_rx_stream::stream_description> rx_stream_description_list;

 @@ bool radio_session_uhd_impl::set_time_to_gps_time()

  if (!device.get_sensor(sensor_name, frac_secs)) {
    fmt::print("Error: not possible to read sensor {}. {}\n", sensor_name, device.get_error_message());
    return false;
  }

+  frac_secs = fmod(frac_secs, 1000000.0);

  // Get time and set
  fmt::print("Setting USRP time to {}s\n", frac_secs);

The UE has attached to the gNb and can connect to the Internet normally.

My radio configuration is the following:

ru_sdr:
  device_driver: uhd                                              
  device_args: type=b200,num_recv_frames=64,num_send_frames=64   
  srate: 23.04 #30.72 #23.04     
  clock: gpsdo #internal #gpsdo
  sync: gpsdo #internal #gpsdo  
  otw_format: sc12
  tx_gain: 79 #50                                                  
  rx_gain: 60 
xavierarteaga commented 5 months ago

Hi @baolindong, Some fixes were included in the previous release that should fix this problem. Is this still happening?