sqlite connection without Database Toolbox

[martin-rdz]

I am trying to get the processing chain running on one of the servers in the containers, but the database drivers cause some issues.

With matlab2014 I get:

>> >> >>
    ____  _                               _____  ____
   / __ \(_)________ _______________     |__  / / __ \
  / /_/ / / ___/ __ `/ ___/ ___/ __ \     /_ < / / / /
 / ____/ / /__/ /_/ (__  |__  ) /_/ /   ___/ // /_/ /
/_/   /_/\___/\__,_/____/____/\____/   /____(_)____/
Undefined function 'database' for input arguments of type 'char'.

Error in testSQLiteJDBC (line 19)
conn = database(dbFile, '', '', 'org:sqlite:JDBC', sprintf('jdbc:sqlite:%s',
dbFile));

Error in initPicassoToolbox (line 48)
if ~ testSQLiteJDBC()

Even when adding the Pollynet_Processing_Chain/include/sqlite-jdbc-3.30.1.jar manually to the javaclasspath.txt, the database() syntax does not work. However, the way described here would work:

jdbc = org.sqlite.JDBC;
props = java.util.Properties;
conn = jdbc.createConnection('jdbc:sqlite:C:\Yair\Data\IGdb 2017-11-13.sqlite',props);  % org.sqlite.SQLiteConnection object
% Prepare and run an SQL query statement
sqlStr = ['select * from data_table where ' conditionStr];
stmt = conn.createStatement;     % org.sqlite.jdbc4.JDBC4Statement object
rs = stmt.executeQuery(sqlStr);  % org.sqlite.jdbc4.JDBC4ResultSet object

For some reason also the addSQLiteJDBC() call did not work.

Any suggestions?

[ZPYin]

Hi Martin,

Thanks for reporting this and very informative feedbacks for diagnosing.

Regarding this issue, I suspect whether Matlab Database Toolbox was well activated for the Matlab 2014 in your container, since database function is a member of this Toolbox.

You can firstly try the command below in the Matlab command line to test whether Database Toolbox was installed.

ver

If Database Toolbox was installed, you can then check whether the path of the toolbox was inserted to Matlab search path correctly. You can find more instructions here.

Best regards, Zhenping

[martin-rdz]

Hi Zhenping,

thanks for your quick response :) This time it's the oceanet container...

Adding the path did not help (had to do it with addpath, as there is no gui available on the server). The toolbox is available with ver, but it cannot be activated as it is not licensed:

>> license('test', 'database')     

ans =

     0

>> license('checkout', 'database')
License checkout failed.
License Manager Error -5
Cannot find a license for database.

Troubleshoot this issue by visiting: 
http://www.mathworks.com/support/lme/R2014a/5

Diagnostic Information:
Feature: database 
License path: ....
Licensing error: -5,357.

ans =

     0

see here: https://de.mathworks.com/matlabcentral/answers/387273-not-able-to-use-installed-matlab-toolbox-image-processing#comment_618828

Best, Martin

[ZPYin]

However, the way described here would work:

It's the first time for me to realize that Matlab Database Toolbox could be an issue to run the processing program. Then it should be worthy of considering to replace the current Database IO operations with the Java functions (as you suggested).

If so, the functions listed below should be updated to be independent of Database Toolbox:

It should also be noticed that mySQL database connector was also used in accessing polly database (in function writeTodoListAuto.m); therefore, alternative for mySQL connector should be also found (here shows a demo for this).

[martin-rdz]

@HolgerPollyNet any chance that someone can take care of that before November?

[ZPYin]

@HolgerPollyNet any chance that someone can take care of that before November?

@HolgerPollyNet I can do this. But this may takes time, since several functions need to updated and tested. I will make it ready before the time that Martin requested.

[ZPYin]

SQLite access with Matlab database toolbox has been replaced with Java DataBase Connector (JDBC).

I have tested the commit https://github.com/PollyNET/Pollynet_Processing_Chain/commit/173df4e294e319abf1636b5263668ea18e24b04b under Matlab 14a (Win64) and it works well. But don't forget to pull the latest commit and test it in the pc that you will use in the near future. And let me know if anything was wrong.

Cheers!

[HolgerPollyNet]

Currently we have on the server VERY OFTEN the following error:

[2022-11-02 16:17:49] --> start saving lidar calibration constants. Error using saveLiConst (line 62) Java exception occurred: org.sqlite.SQLiteException: [SQLITE_ERROR] SQL error or missing database (no such column: NaN) at org.sqlite.core.DB.newSQLException(DB.java:941) at org.sqlite.core.DB.newSQLException(DB.java:953) at org.sqlite.core.DB.throwex(DB.java:918) at org.sqlite.core.NativeDB._exec_utf8(Native Method) at org.sqlite.core.NativeDB._exec(NativeDB.java:94) at org.sqlite.jdbc3.JDBC3Statement.executeUpdate(JDBC3Statement.java:109)

Error in picassoProcV3 (line 4389) saveLiConst(dbFile, LC.LC_raman_355, LC.LCStd_raman_355, ...

Error in picassoProcTodolist (line 62) reportTmp = picassoProcV3(pollyDataFile, pollyDataTasks.pollyType{iTask}, ...

Error in picassoProcHistoryData (line 69) picassoProcTodolist(PicassoConfigFile);

Is this related to recent developments? @ZPYin @martin-rdz @ulysses78 I will make a hotfix on the server using the try and catch function of matlab, but this will be not sustainable. So we need to keep this in mind.

[ZPYin]

I checked through the processing log files on 2022-11-02, but I didn't find the error as you mentioned above. ** Could you provide the log file? It would be helpful for me to re-produce the error.

@HolgerPollyNet

[HolgerPollyNet]

@ZPYin There is for example a log file called: log_push_current_pollydata - dwd_not_working

its a copy of a recent problem. here it occurs at line 1568 with dwd data.

But if you want to re-produce the problem, you need to use the config pollyxt_dwd_config_20221001.json.bak or pollyxt_dwd_config.json instead of pollyxt_dwd_config_20221001.json.

In the later one, I set all flags to false to allow the running of the chain.

[ZPYin]

I think it can be closed now. (see bugfix above)

[HolgerPollyNet]

Thanks @ZPYin !!

I currently try to pull it to the server, but I have problems:

fatal: Not possible to fast-forward, aborting.

Shall I stash the changes? Could you have a look? After weekend is fine as well ;-)

[ZPYin]

I have merged the commit into the server. 🍻

Regarding the error of git merge, I think it was caused by the parallel file editing in the server, while I edited the same file (picassoProcV3.m) for the bug fix. This error message is helpful to inform some potential conflicts after merging.

I used the command below to rebase the HEAD and merge the branch

git pull --rebase origin dev

And didn't find any conflicts... (Good news). Maybe this is just a simple warning...

[HolgerPollyNet]

Unfortunately, one error gone the other one occurs:

Error using internal.matlab.imagesci.nc/openToRead (line 1259) Could not open /.../2022_11_11_Fri_1v2_06_00_01.nc for reading.

Error in internal.matlab.imagesci.nc (line 121) this.openToRead();

Any idea?

[HolgerPollyNet]

Temporary solved by setting:

"flagDeleteData": false,

But I guess it should in principle have nothing to do with the recent commit?!?

[ZPYin]

Unfortunately, one error gone the other one occurs:

Error using internal.matlab.imagesci.nc/openToRead (line 1259) Could not open /.../2022_11_11_Fri_1v2_06_00_01.nc for reading.

Error in internal.matlab.imagesci.nc (line 121) this.openToRead();

Any idea?

It seems like that this file (2022_11_11_Fri_1v2_06_00_01.nc) was opened by other programs and still hanged somewhere when Picasso tried to delete this file. I checked the todofile_list just now, and didn't find this file anymore. So this error should not occur at the moment.