daphne-eu / daphne

DAPHNE: An Open and Extensible System Infrastructure for Integrated Data Analysis Pipelines
Apache License 2.0
68 stars 62 forks source link

PCA script runs with no output #573

Closed aristotelis96 closed 1 year ago

aristotelis96 commented 1 year ago

Running Principal Component Analysis (PCA) ./bin/daphne scripts/algorithms/pca.daph X=\"data/lmDS/wine.csv\" K=2 center=true scale=true currently displays the following error:

JIT session error: Symbols not found: [ _ctable__DenseMatrix_double__DenseMatrix_double__DenseMatrix_size_t__double__int64_t__int64_t ]
JIT-Engine invocation failed: Failed to materialize symbols: { (main, { _mlir_main, main, _mlir__mlir_ciface_main, _mlir_ciface_main }) }Program aborted due to an unhandled Error:
Failed to materialize symbols: { (main, { _mlir_main, main, _mlir__mlir_ciface_main, _mlir_ciface_main }) }

The above error can easily be bypassed by casting the DenseMatrix<size_t> to DenseMatrix<double>: diagmat = ctable(seq(1.0, N, 1.0), as.f64(decreasing_Idx));

However, after the fix the script runs without displaying any output or error.

*The generated log files from the newly implemented logger are empty.

corepointer commented 1 year ago

Currently we have not the best defaults for the logger - it's OFF. Edit the log level limit in UserConfig.json and add --config UserConfig.json to our DAPHNE invocation.

As for the initial error message: as you might have already found out (because you correctly casted the inputs), the operation is not implemented of the combination that was initially used with the size_t.

aristotelis96 commented 1 year ago

Thank you @corepointer for the response. I did provide the config file as you suggested, however I did not change the log level limit. :)

The logs now indicate an issue with the matrix dimensions used for ctable which is a bug on the script itself (pca.daph) rather than the daphne system. I think we can keep this issue open until the pca.daph is fixed.

pdamme commented 1 year ago

Indeed, there were a few little bugs in the PCA script itself, and I have just fixed them. It should run smoothly now, both with and without --vec.

Nevertheless, the exception thrown by the buggy script didn't result in any message printed to the user. Instead, the script simply failed silently. During development, it is common to have a few little bugs in a DaphneDSL script, and then (good) error messages are very important. However, this is a separate issue we should discuss in the near future.