Working with a large dataset

[keyvan-m-sadeghi]

Hi @jmmcd, etc.,

I'm hoping to get ffx working on a dataset with 10 million rows, 26 features.

I ran to a few issues which I summarised below:

• The time complexity of the algorithm, is it exponential? For 100 rows it takes 25 seconds, 37s for 200 rows and 163s for 300 rows.

• I'm getting a new warning:

  /home/keyvan/Git/ffx/ffx/core.py:568: RuntimeWarning: invalid value encountered in double_scalars
  minx + 0.2*rangex, maxx - 0.2*rangex + 0.1*rangex, stepx)

• I see some names in the formulas that I don't have, for example, I do have variables named 'm' and 'o', but I get things like 'm3', 'm8' and 'o1' in the formulas:

  0.510
  0.296 + 0.473*m3
  0.503 + 0.171*m3 - 0.163*max(0,0.867-m3) + 0.117*max(0,m3-0.200) - 0.101*max(0,0.733-m3) + 0.0163*max(0,m3-0.333)
  (0.186 + 0.243*m3 + 0.242*m3 + 0.198*m5 * p) / (1.0 - 0.0998*m3 - 0.0995*m3 - 0.0774*p)
  (0.178 + 0.250*m3 + 0.249*m3 + 0.235*m5 * p) / (1.0 - 0.0990*m3 - 0.0988*m3 - 0.0779*p - 0.0345*o1 * p)
  (0.142 + 0.403*m5 * p + 0.286*m3 + 0.286*m3) / (1.0 - 0.197*o1 * p - 0.0944*m3 - 0.0934*m3 - 0.0888*m8 - 0.0728*p)
  Performance: 163.948806047 seconds

  what do these mean?

Cheers, K

[jmmcd]

The relationship (for FFX) between runtime and number of rows is probably determined by the same relationship for ElasticNet.

For the others, do you see these errors with (say) a 100-row dataset? If so can you attach some minimal code?

For m3 etc is there any chance a "^" character is being stripped by your terminal or something like that?

[jmmcd]

Please re-open if you have a minimal working example so we can replicate.