time resolution requirement

[wreckdump]

Is there a suggested time resolution requirement for tracking?

The TOBAC paper (https://gmd.copernicus.org/articles/12/4551/2019/) only covers up to 30 minute resolution, but what I have is 3 hourly dataset. Would this not work?

[w-k-jones]

The scale dependency in tobac depends on how far each feature moves between time steps versus the separation between neighbouring features. The shorter the distance objects move between time steps, compared to the distance between objects, the more reliable tracking will be. Predictive tracking helps a bit here, but it needs to work out the initial motion to get started.

So for your case, if the objects you want to track are widely spaced (such as MCSs) or slowly moving, tobac should work ok.

(note that this scaling with motion vs horizontal length scales is a challenge that all tracking algorithms face, with some methods working better than others for different scales. It's something we want to try and improve further in tobac)

[wreckdump]

Thank you for your reply.

I was just wondering why I would get the error message of

UserWarning: Could not generate velocity field for prediction: no tracks warn('Could not generate velocity field for prediction: no tracks')

during the linking process which I thought it might be related to the time resolution issue.

[JuliaKukulies]

@wreckdump, this is a warning from trackpy (the library our tracking is based on) when no tracks are found in the predictive tracking. You are right, this could be a time resolution issue, but like @w-k-jones said, that will depend a lot on the features that you detected in the first step and on what kind of phenomenon you track.

Could you post your feature detection output dataframe and tell us a bit more about the atmospheric field you track on and which parameters you used for feature detection and tracking? That might help to give you a hint on why the tracking step does not produce any output for you.

[wreckdump]

Thank you for the reply.

For the feature detection pandas dataframe, I saved it in h5 file, like how it was done in the example. It looks quite long. Do you want me to post the entire thing?

I am using brightness temperature field from a model which has 0.25 degree spatial resolution, and 3 hourly temporal resolution with 312 time steps. And I am using a single brightness temperature threshold of 240 K. I am looking to track the features that are below the threshold.

Here are some of the specified options in my code

dxy,dt      =  0.25*111.0e3, 3*3600.0 # 0.25 degree in km, and 3 hourly in seconds

param_fet={}
param_fet['target']='minimum'
param_fet['threshold']=[240]
param_fet['n_min_threshold']=4
param_fet['position_threshold']='weighted_diff'
param_fet['sigma_threshold']=0.5

fet               =  tobac.feature_detection.feature_detection_multithreshold(tb, dxy, **param_fet)

[JuliaKukulies]

Yes, I think it would be helpful to have a look at your feature detection output. You could, for example, attach it here.

It depends, of course, on the region, season and model you use to track, but given that n_min_threshold is rather small for a brightness temperature threshold of 240K, I would expect many small-scale (and not widely spaced) cloud systems that potentially move fast or disappear within 3 hours to be detected in addition to MCSs.

[wreckdump]

Okay. Here is the first 50 line of the feature detection pandas dataframe.

       frame  idx      hdim_1       hdim_2   num  threshold_value  feature                 time              timestr   latitude   longitude
0          0    1   22.100121   215.379521   695              240        1  2020-01-21 00:00:00  2020-01-21 00:00:00  24.474970 -126.155120
1          0    3    5.549077   589.025375    36              240        2  2020-01-21 00:00:00  2020-01-21 00:00:00  28.612731  -32.743656
2          0    4    1.888477  1104.101555    40              240        3  2020-01-21 00:00:00  2020-01-21 00:00:00  29.527881   96.025389
3          0    7    6.321160  1062.414647     8              240        4  2020-01-21 00:00:00  2020-01-21 00:00:00  28.419710   85.603662
4          0    9    8.811247  1115.964015     8              240        5  2020-01-21 00:00:00  2020-01-21 00:00:00  27.797188   98.991004
5          0   10   18.908375   207.624846    45              240        6  2020-01-21 00:00:00  2020-01-21 00:00:00  25.272906 -128.093789
6          0   11   24.586862   224.727638    12              240        7  2020-01-21 00:00:00  2020-01-21 00:00:00  23.853284 -123.818091
7          0   13   48.265269   156.876361   145              240        8  2020-01-21 00:00:00  2020-01-21 00:00:00  17.933683 -140.780910
8          0   22   63.533779   171.233289     6              240        9  2020-01-21 00:00:00  2020-01-21 00:00:00  14.116555 -137.191678
9          0   23   72.150873   188.876996    91              240       10  2020-01-21 00:00:00  2020-01-21 00:00:00  11.962282 -132.780751
10         0   25   72.388204   176.031177    17              240       11  2020-01-21 00:00:00  2020-01-21 00:00:00  11.902949 -135.992206
11         0   28   91.775584  1418.921023   418              240       12  2020-01-21 00:00:00  2020-01-21 00:00:00   7.056104  174.730256
12         0   29   83.989559   224.817098    74              240       13  2020-01-21 00:00:00  2020-01-21 00:00:00   9.002610 -123.795725
13         0   30   89.755968   387.181178   268              240       14  2020-01-21 00:00:00  2020-01-21 00:00:00   7.561008  -83.204705
14         0   31   84.879717   140.486537    12              240       15  2020-01-21 00:00:00  2020-01-21 00:00:00   8.780071 -144.878366
15         0   32   85.179920  1433.668502    17              240       16  2020-01-21 00:00:00  2020-01-21 00:00:00   8.705020  178.417126
16         0   34   88.038070   416.329671    74              240       17  2020-01-21 00:00:00  2020-01-21 00:00:00   7.990483  -75.917582
17         0   36   87.738341    29.450031     5              240       18  2020-01-21 00:00:00  2020-01-21 00:00:00   8.065415 -172.637492
18         0   38   89.136136  1398.555896    10              240       19  2020-01-21 00:00:00  2020-01-21 00:00:00   7.715966  169.638974
19         0   39   90.473231   207.199987    22              240       20  2020-01-21 00:00:00  2020-01-21 00:00:00   7.381692 -128.200003
20         0   42   94.098800   168.161555    48              240       21  2020-01-21 00:00:00  2020-01-21 00:00:00   6.475300 -137.959611
21         0   43   96.530685   399.605554    36              240       22  2020-01-21 00:00:00  2020-01-21 00:00:00   5.867329  -80.098612
22         0   45   92.965907   968.252976    11              240       23  2020-01-21 00:00:00  2020-01-21 00:00:00   6.758523   62.063244
23         0   47   97.111180   355.787793    21              240       24  2020-01-21 00:00:00  2020-01-21 00:00:00   5.722205  -91.053052
24         0   48   97.607752     4.788627    60              240       25  2020-01-21 00:00:00  2020-01-21 00:00:00   5.598062 -178.802843
25         0   49   95.359886    23.768679    13              240       26  2020-01-21 00:00:00  2020-01-21 00:00:00   6.160028 -174.057830
26         0   50   97.860021  1115.095093    13              240       27  2020-01-21 00:00:00  2020-01-21 00:00:00   5.534995   98.773773
27         0   51   98.813884    15.794988     6              240       28  2020-01-21 00:00:00  2020-01-21 00:00:00   5.296529 -176.051253
28         0   52   99.376609    72.394128     6              240       29  2020-01-21 00:00:00  2020-01-21 00:00:00   5.155848 -161.901468
29         0   54  103.904307  1062.799459   132              240       30  2020-01-21 00:00:00  2020-01-21 00:00:00   4.023923   85.699865
30         0   55  100.106617  1085.445919     5              240       31  2020-01-21 00:00:00  2020-01-21 00:00:00   4.973346   91.361480
31         0   57  102.598178    49.964839     9              240       32  2020-01-21 00:00:00  2020-01-21 00:00:00   4.350456 -167.508790
32         0   58  106.202123  1119.308606    50              240       33  2020-01-21 00:00:00  2020-01-21 00:00:00   3.449469   99.827152
33         0   62  111.538887  1041.088970    71              240       34  2020-01-21 00:00:00  2020-01-21 00:00:00   2.115278   80.272242
34         0   63  117.493482   137.933757    89              240       35  2020-01-21 00:00:00  2020-01-21 00:00:00   0.626630 -145.516561
35         0   64  116.574068   407.999268    84              240       36  2020-01-21 00:00:00  2020-01-21 00:00:00   0.856483  -78.000183
36         0   65  111.000000   423.729660     5              240       37  2020-01-21 00:00:00  2020-01-21 00:00:00   2.250000  -74.067585
37         0   66  117.079201  1259.648283    45              240       38  2020-01-21 00:00:00  2020-01-21 00:00:00   0.730200  134.912071
38         0   69  116.486733  1154.382695    64              240       39  2020-01-21 00:00:00  2020-01-21 00:00:00   0.878317  108.595674
39         0   73  122.893065  1124.717396    38              240       40  2020-01-21 00:00:00  2020-01-21 00:00:00  -0.723266  101.179349
40         0   75  121.098875   427.551072    10              240       41  2020-01-21 00:00:00  2020-01-21 00:00:00  -0.274719  -73.112232
41         0   79  125.079295   497.188594    13              240       42  2020-01-21 00:00:00  2020-01-21 00:00:00  -1.269824  -55.702852
42         0   81  129.157636   427.624584    10              240       43  2020-01-21 00:00:00  2020-01-21 00:00:00  -2.289409  -73.093854
43         0   83  132.903691   468.400394    52              240       44  2020-01-21 00:00:00  2020-01-21 00:00:00  -3.225923  -62.899902
44         0   85  138.977488   448.620865   146              240       45  2020-01-21 00:00:00  2020-01-21 00:00:00  -4.744372  -67.844784
45         0   86  137.093751  1324.585016    64              240       46  2020-01-21 00:00:00  2020-01-21 00:00:00  -4.273438  151.146254
46         0   90  133.641393   769.942807     5              240       47  2020-01-21 00:00:00  2020-01-21 00:00:00  -3.410348   12.485702
47         0   91  138.434300  1353.069119   133              240       48  2020-01-21 00:00:00  2020-01-21 00:00:00  -4.608575  158.267280
48         0   94  147.652475   481.180371   136              240       49  2020-01-21 00:00:00  2020-01-21 00:00:00  -6.913119  -59.704907

Region is tropics (30N to 30S and 180W to 180E), and the date range is 2020-01-21 to 2020-02-28. The model is global storm resolving models (https://www.esiwace.eu/the-project/past-phases/dyamond-initiative). Their outputs are remapped down to 0.25 degree. Just eyeballing the field thresholded by 240 K, it seems that the features are not that far off in between each 3 hour time steps.

[JuliaKukulies]

As @w-k-jones explained, the short horizontal length scale and small distance between these tropical systems in relation to their moving distance and probably relatively short lifetime could be the cause of the failed linking. Is it possible attach the file rather than posting the output, than I could look at it in more detail?

[wreckdump]

out.txt

Here is the entire output of feature detection.

[JuliaKukulies]

Thanks! So based on this feature dataframe, I am getting a result for the tracking. What were your tracking parameters and what version of tobac are you using?

[wreckdump]

I am using tobac version 1.4.1, which I got it via pip.

Here are the tracking parameters

For feature detection

param_fet['target']='minimum'
param_fet['threshold']=[240]
param_fet['n_min_threshold']=4
param_fet['position_threshold']='weighted_diff'
param_fet['sigma_threshold']=0.5

For segmentation

param_seg['target']='minimum'
param_seg['method']='watershed'
param_seg['threshold']=240

For linking

param_lnk['v_max']=None
param_lnk['d_max']=111*10
param_lnk['stubs']=1
param_lnk['order']=2
param_lnk['extrapolate']=0
param_lnk['memory']=0
param_lnk['adaptive_stop']=0.2
param_lnk['adaptive_step']=0.95
param_lnk['subnetwork_size']=100
param_lnk['method_linking']= 'random'

This is my first time using the package, so I am baiscally trying it out. What I am looking to do is to track any 'clusters' that has values below brightness temperature of ~240 K within 10 degrees radius (or 10 deg. x 10 deg. box to be exact).

[w-k-jones]

The value for d_max looks quite small here. It's in units of meters so perhaps if you multiply that value by a thousand you'll get the result that you're looking for?

[wreckdump]

The value for d_max looks quite small here. It's in units of meters so perhaps if you multiply that value by a thousand you'll get the result that you're looking for?

That's a good point. I totally missed that. I'll try that.

[wreckdump]

Um... the code is stuck, its more than 8 hours now, at the second time step for the linking process. Is this normal?

trackpy.linking.linking.link_iter:  Frame 1: 137 trajectories present.
INFO:trackpy.linking.linking:Frame 1: 137 trajectories present.
trackpy.linking.linking.link_iter:  Frame 2: 153 trajectories present.
INFO:trackpy.linking.linking:Frame 2: 153 trajectories present.

[freemansw1]

Um... the code is stuck, its more than 8 hours now, at the second time step for the linking process. Is this normal?

trackpy.linking.linking.link_iter:  Frame 1: 137 trajectories present.
INFO:trackpy.linking.linking:Frame 1: 137 trajectories present.
trackpy.linking.linking.link_iter:  Frame 2: 153 trajectories present.
INFO:trackpy.linking.linking:Frame 2: 153 trajectories present.

I've noticed that this can sometimes happen with trackpy, but I can't seem to replicate well. Try reducing your subnetwork_size.

[wreckdump]

Um... the code is stuck, its more than 8 hours now, at the second time step for the linking process. Is this normal?

trackpy.linking.linking.link_iter:  Frame 1: 137 trajectories present.
INFO:trackpy.linking.linking:Frame 1: 137 trajectories present.
trackpy.linking.linking.link_iter:  Frame 2: 153 trajectories present.
INFO:trackpy.linking.linking:Frame 2: 153 trajectories present.

I've noticed that this can sometimes happen with trackpy, but I can't seem to replicate well. Try reducing your subnetwork_size.

Yep. Adjusting subnetwork_size seems to solve this issue. Thank you.