search

openml / OpenML

Open Machine Learning
https://openml.org
BSD 3-Clause "New" or "Revised" License
662 stars 91 forks source link

Unable to install OpenML locally #560

Closed joaquinvanschoren closed 5 years ago

joaquinvanschoren commented 6 years ago

From @rkadlec on May 24, 2017 15:29

I wanted to try OpenML locally therefore I followed instructions at https://github.com/openml/OpenML/wiki/Local-Installation

After editing website/openml_OS/config/BASE_CONFIG.php I tried to test the local install, however I see just blank page. Interesting thing is that it doesn't matter what configuration values do I use. Even when I try complete nonsense values I don't get any error, just the blank page.

Is this a symptom of something that sounds familiar to you?

Copied from original issue: openml/website#143

joaquinvanschoren commented 6 years ago

From @haps-basset on June 6, 2017 1:7

I had similar problem If your installation was done on Centos/RHEL platform then execute the following command: restorecon -R -v /var/www/

Haps

joaquinvanschoren commented 6 years ago

From @rkadlec on June 6, 2017 16:5

Thank you for your reply, unfortunately this didn't help. I still get blank page, there is no HTML inside. I should note that I am running other php apps on the same server without any issues.

joaquinvanschoren commented 6 years ago

From @haps-basset on June 6, 2017 22:32

Sorry, What response are you getting sending URL as follows http:/your_server/index.php? You mentioned that your server has many php applications so I am guessing that those apps have dedicated virtual directors, could you confirm? I have notice that openml server does not like to be installed under some virtual director structure and must be hosted by the “root” of /var/www/html.

joaquinvanschoren commented 6 years ago

From @rkadlec on June 8, 2017 8:54

Great. I am slowly progressing. I moved OpenML to /var/www/html and after fixing several minor config issues I am in a state where I can see the initial page, however when I open "Data" page I get an error complaining about missing Elasticsearch index. https://github.com/openml/OpenML/wiki/Local-Installation says: "you'll also want to build your own search indices to show all data on the website" however it doesn't give any advice on how to create them.

Here is Elasticsearch exception that I get:

    An uncaught Exception was encountered

    Type: Elasticsearch\Common\Exceptions\Missing404Exception

    Message: {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index","resource.type":"index_expression","resource.id":"openml","index":"openml"}],"type":"index_not_found_exception","reason":"no such index","resource.type":"index_expression","resource.id":"openml","index":"openml"},"status":404}

    Filename: /var/www/html/openml_OS/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Connections/Connection.php

    Line Number: 590

    Backtrace:

    File: /var/www/html/openml_OS/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Connections/Connection.php
    Line: 261
    Function: process4xxError

    File: /var/www/html/openml_OS/vendor/react/promise/src/FulfilledPromise.php
    Line: 25
    Function: Elasticsearch\Connections\{closure}

    File: /var/www/html/openml_OS/vendor/guzzlehttp/ringphp/src/Future/CompletedFutureValue.php
    Line: 55
    Function: then

    File: /var/www/html/openml_OS/vendor/guzzlehttp/ringphp/src/Core.php
    Line: 341
    Function: then

    File: /var/www/html/openml_OS/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Connections/Connection.php
    Line: 282
    Function: proxy

    File: /var/www/html/openml_OS/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Connections/Connection.php
    Line: 159
    Function: Elasticsearch\Connections\{closure}

    File: /var/www/html/openml_OS/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Transport.php
    Line: 107
    Function: performRequest

    File: /var/www/html/openml_OS/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Client.php
    Line: 1346
    Function: performRequest

    File: /var/www/html/openml_OS/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Client.php
    Line: 179
    Function: performRequest

    File: /var/www/html/openml_OS/views/pages/frontend/search/subpage/data.php
    Line: 9
    Function: get

    File: /var/www/html/openml_OS/helpers/cms_helper.php
    Line: 40
    Function: view

    File: /var/www/html/openml_OS/views/pages/frontend/search/body.php
    Line: 16
    Function: subpage

    File: /var/www/html/openml_OS/helpers/cms_helper.php
    Line: 19
    Function: view

    File: /var/www/html/openml_OS/helpers/cms_helper.php
    Line: 23
    Function: loadpage

    File: /var/www/html/openml_OS/views/frontend_main.php
    Line: 312
    Function: body

    File: /var/www/html/openml_OS/controllers/Frontend.php
    Line: 95
    Function: view

    File: /var/www/html/index.php
    Line: 334
    Function: require_once
joaquinvanschoren commented 6 years ago

Hi, sorry I'm snowed under right now, so I can't really do much testing.

What you basically need to do is install Elasticsearch 5, and then update the BASE_CONFIG.php file with the ElasticSearch endpoint (ES_URL), by default that is something like localhost:9200, but you'll probably want to set it up properly (e.g. use a protected proxy).

define( 'ES_URL', 'https://www.yourserver.org:port' ); define( 'ES_USERNAME', 'user' ); define( 'ES_PASSWORD', 'pass' );

You then need to create a new index called 'openml' (see ElasticSearch docs on how to do that). Finally, you need to build your mappings. There is a command line interface for this, which you need to run from the root folder (where index.php resides). The command is php index.php cron index "type" id

where type is the type of the mapping, so you need to run this for all mappings: task, flow, data, measure, task_type, run, study, like, download, downvote

and 'id' is the (optional) starting id, i.e. 'index run 1' will index all runs starting from run 1. This will first initialize the mappings and then start indexing all objects in the openml_expdb database. Note: if you have downloaded the full OpenML database (the nightly snapshot), it will take a significant amount of time to index all runs.

We do have a webinterface for the search engine, but this is completely undocumented, unpolished, and only for admin use. Email me if you need more details.

Finally, I'm very interested in your use case. Why do you need a local setup of OpenML? Is this just to play around with it or do you have more ambitious goals? It would also be great if one of you could create a docker image with the complete setup, so that other people don't have to go through the complete process again. And finally, if you want to contribute to OpenML development, e.g. by building the docker image or otherwise making setup easier, we would be really happy with that!

joaquinvanschoren commented 6 years ago

From @rkadlec on June 8, 2017 12:10

Thank you, this helped me to do one more step.

I issued these commands that build the index and mappings:

curl -XPUT 'localhost:9200/openml?pretty' -H 'Content-Type: application/json' -d' { "settings" : { "number_of_shards" : 3, "number_of_replicas" : 2 } } '

for TYPE in task flow data measure task_type run study like download downvote do sudo php index.php cron index $TYPE 1 done

and now I get different exception

    An uncaught Exception was encountered

    Type: Elasticsearch\Common\Exceptions\Missing404Exception

    Message: {"_index":"openml","_type":"measure","_id":"NumberOfInstances","found":false}

    Filename: /var/www/html/openml_OS/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Connections/Connection.php

    Line Number: 590

    Backtrace:

    File: /var/www/html/openml_OS/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Connections/Connection.php
    Line: 261
    Function: process4xxError

    File: /var/www/html/openml_OS/vendor/react/promise/src/FulfilledPromise.php
    Line: 25
    Function: Elasticsearch\Connections\{closure}

    File: /var/www/html/openml_OS/vendor/guzzlehttp/ringphp/src/Future/CompletedFutureValue.php
    Line: 55
    Function: then

    File: /var/www/html/openml_OS/vendor/guzzlehttp/ringphp/src/Core.php
    Line: 341
    Function: then

    File: /var/www/html/openml_OS/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Connections/Connection.php
    Line: 282
    Function: proxy

    File: /var/www/html/openml_OS/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Connections/Connection.php
    Line: 159
    Function: Elasticsearch\Connections\{closure}

    File: /var/www/html/openml_OS/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Transport.php
    Line: 107
    Function: performRequest

    File: /var/www/html/openml_OS/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Client.php
    Line: 1346
    Function: performRequest

    File: /var/www/html/openml_OS/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Client.php
    Line: 179
    Function: performRequest

    File: /var/www/html/openml_OS/views/pages/frontend/search/subpage/data.php
    Line: 9
    Function: get

    File: /var/www/html/openml_OS/helpers/cms_helper.php
    Line: 40
    Function: view

    File: /var/www/html/openml_OS/views/pages/frontend/search/body.php
    Line: 16
    Function: subpage

    File: /var/www/html/openml_OS/helpers/cms_helper.php
    Line: 19
    Function: view

    File: /var/www/html/openml_OS/helpers/cms_helper.php
    Line: 23
    Function: loadpage

    File: /var/www/html/openml_OS/views/frontend_main.php
    Line: 312
    Function: body

    File: /var/www/html/openml_OS/controllers/Frontend.php
    Line: 95
    Function: view

    File: /var/www/html/index.php
    Line: 334
    Function: require_once

I want to install OpenML to see how easily it can be extended. I agree that docker image would be helpful :-) If I succeed installing OpenML locally I would consider creating it.

joaquinvanschoren commented 6 years ago

This error says that the 'measure' mapping is missing. As indicated above, you must first create it using the command line: php index.php cron index "mapping"

Same for all other types (data, task, run,...)

joaquinvanschoren commented 6 years ago

From @rkadlec on June 9, 2017 8:4

Hi, I am still struggling with mappings. I am using the latest Elasticsearch 5.4.1. I tried two things:

1) when I use database schema without any data for openml_expdb (https://www.openml.org/downloads/openml_expdb.sql) I can run the command for building mappings without any error, however when I check elasticsearch with "curl -XGET 'localhost:9200/openml/_mapping?pretty'" it doesn't show any mappings.

2) when I use snapshot DB (https://www.openml.org/downloads/ExpDB_SNAPSHOT.sql.gz) there are some errors when the mappings are build:

A PHP Error was encountered

Severity: Notice Message: Use of undefined constant JSON_PRESERVE_ZERO_FRACTION - assumed 'JSON_PRESERVE_ZERO_FRACTION' Filename: /var/www/html/openml_OS/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Serializers/SmartSerializer.php Line Number: 30

Backtrace: File: /var/www/html/openml_OS/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Serializers/SmartSerializer.php Line: 30 Function: _error_handler

    File: /var/www/html/openml_OS/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Endpoints/Bulk.php
    Line: 40
    Function: serialize

    File: /var/www/html/openml_OS/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Client.php
    Line: 644
    Function: setBody

    File: /var/www/html/openml_OS/libraries/ElasticSearch.php
    Line: 434
    Function: bulk

    File: /var/www/html/openml_OS/libraries/ElasticSearch.php
    Line: 362
    Function: index_downvote

    File: /var/www/html/openml_OS/controllers/Cron.php
    Line: 50
    Function: index

    File: /var/www/html/index.php
    Line: 334
    Function: require_once

A PHP Error was encountered

Severity: Warning Message: json_encode() expects parameter 2 to be long, string given Filename: /var/www/html/openml_OS/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Serializers/SmartSerializer.php Line Number: 30

Backtrace: File: /var/www/html/openml_OS/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Serializers/SmartSerializer.php Line: 30 Function: json_encode

    File: /var/www/html/openml_OS/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Endpoints/Bulk.php
    Line: 40
    Function: serialize

    File: /var/www/html/openml_OS/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Client.php
    Line: 644
    Function: setBody

    File: /var/www/html/openml_OS/libraries/ElasticSearch.php
    Line: 434
    Function: bulk

    File: /var/www/html/openml_OS/libraries/ElasticSearch.php
    Line: 362
    Function: index_downvote

    File: /var/www/html/openml_OS/controllers/Cron.php
    Line: 50
    Function: index

    File: /var/www/html/index.php
    Line: 334
    Function: require_once
joaquinvanschoren commented 6 years ago

From @haps-basset on June 12, 2017 23:25

I am following your instruction with limited success. So far I was able to create index and mapping as you described in previous posts. I had similar problem with re-creating databases using openml snap-shot but openml_expdb.sql worked quite well for me thus I ma not sure ... My current problem is that I cannot add data, tasks etc. in theory I should be pointed to add .. page but I ma geting blank page. To my surprise firefox debugger shows all http transaction with status 200 (ok) and console message “All is well”, from ping to elaseticsearch? I running out of ideas how install this apps.

Haps

PS. I am happy to send you screen-dumps form firefox if you wish.

joaquinvanschoren commented 6 years ago

@haps-basset Is it the upload form that is blank? Did you configure php to show errors and warnings?

joaquinvanschoren commented 6 years ago

@rkadlec Are you still having difficulties creating the mappings? I googled your first error. Does this help? Is may be a matter of using the right PHP and ES versions https://github.com/elastic/elasticsearch-php/issues/534

If it helps, our latest install uses ES 5.0.2 and PHP 7.0.15.

joaquinvanschoren commented 6 years ago

I updated some code to automatically build the indices from scratch. It's in the develop branch but will likely be pushed to master soon. Below is a .sh script that should do everything, assuming that ElasticSearch and MySQL are installed and configured. Let me know how it goes.

# reset ES search index
echo "Deleting and recreating the ES index: "
curl -XDELETE http://localhost:9200/openml
curl -XPUT 'localhost:9200/openml?pretty' -H 'Content-Type: application/json' -d'
{
    "settings" : {
        "index" : {
            "number_of_shards" : 3,
            "number_of_replicas" : 2
        }
    }
}
'

# go to right directory
cd /var/www/openml.org/public_html/

# import database
# Download from http://openml.org/downloads/openml_expdb.sql
mysql -u root -pPASS < openml_expdb.sql

# rebuild search index
sudo php index.php cron build_es_indices
joaquinvanschoren commented 6 years ago

You also need to set up cronjobs for the OpenML services:

java -Xmx4G -jar /var/www/openml.org/public_html/openml_OS/third_party/OpenML/Java/evaluate.jar -config "server = http://yourserver.org/;api_key =XXX" -f process_dataset &> /dev/null

There are 3 main services:

joaquinvanschoren commented 6 years ago

From @rkadlec on June 15, 2017 9:43

Here is my update: 1) I upgraded PHP from 5.4 to 7.x 2) I started using develop branch

I get new exception that is probably caused by our network environment, we are behind a proxy. When I run "sudo php index.php cron build_es_indices" I get an error from the proxy even though ES is running on localhost and I have $no_proxy env variable properly set for localhost. I tried setting EXTERNAL_API_PROXY* properties in BASE_CONFIG.php but this doesn't change anything.
Here is the complete error:

Starting downvote indexer from id 1... An uncaught Exception was encountered

Type: Elasticsearch\Common\Exceptions\ServerErrorResponseException Message: <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">

ERROR: The requested URL could not be retrieved

ERROR

The requested URL could not be retrieved

The following error was encountered while trying to retrieve the URL: http://localhost:9200/openml/_mapping/downvote

Connection to ::1 failed.

The system returned: (111) Connection refused

The remote host or network may be down. Please try the request again.

Your cache administrator is webmaster.


Filename: /var/www/html/openml_OS/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Connections/Connection.php Line Number: 686

Backtrace: File: /var/www/html/openml_OS/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Connections/Connection.php Line: 664 Function: tryDeserializeError

File: /var/www/html/openml_OS/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Connections/Connection.php
Line: 625
Function: tryDeserialize500Error

File: /var/www/html/openml_OS/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Connections/Connection.php
Line: 264
Function: process5xxError

File: /var/www/html/openml_OS/vendor/react/promise/src/FulfilledPromise.php
Line: 25
Function: Elasticsearch\Connections\{closure}

File: /var/www/html/openml_OS/vendor/guzzlehttp/ringphp/src/Future/CompletedFutureValue.php
Line: 55
Function: then

File: /var/www/html/openml_OS/vendor/guzzlehttp/ringphp/src/Core.php
Line: 341
Function: then

File: /var/www/html/openml_OS/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Connections/Connection.php
Line: 282
Function: proxy

File: /var/www/html/openml_OS/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Connections/Connection.php
Line: 159
Function: Elasticsearch\Connections\{closure}

File: /var/www/html/openml_OS/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Transport.php
Line: 106
Function: performRequest

File: /var/www/html/openml_OS/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Namespaces/AbstractNamespace.php
Line: 72
Function: performRequest

File: /var/www/html/openml_OS/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Namespaces/IndicesNamespace.php
Line: 286
Function: performRequest

File: /var/www/html/openml_OS/libraries/ElasticSearch.php
Line: 381
Function: getMapping

File: /var/www/html/openml_OS/controllers/Cron.php
Line: 69
Function: index_from

File: /var/www/html/openml_OS/controllers/Cron.php
Line: 81
Function: indexfrom

File: /var/www/html/index.php
Line: 334
Function: require_once
joaquinvanschoren commented 6 years ago

From @haps-basset on June 16, 2017 4:5

@joaquinvanschoren

According to instruction, I enabled debugging as follows but I am getting "blank" screen no errors etc. I am getting most information form browser debug mode . It looks to me that the elastic-search works for me at least external tests show that is ok

define( 'DEBUG', true );

joaquinvanschoren commented 6 years ago

From @haps-basset on June 17, 2017 7:30

@joaquinvanschoren

Small progress but still plenty problems. I am getting the following message/error from any attempt to “share” pages but I cannot find the reason of this problem Any ideas?

'div class="alert alert-dismissible alert-error"> JavaScript is required to properly view the contents of this page! </div'

joaquinvanschoren commented 6 years ago

From @haps-basset on June 22, 2017 5:30

After fixing few issues with system configuration, now I am able to select "new data" from menu and I am getting the following error message?

A PHP Error was encountered Severity: Notice Message: Undefined property: stdClass::$ttid Filename: new/pre.php Line Number: 20 Backtrace: File: /var/www/html/openml_OS/views/pages/frontend/new/pre.php Line: 20 Function: _error_handler File: /var/www/html/openml_OS/helpers/cms_helper.php Line: 19 Function: view File: /var/www/html/openml_OS/controllers/Frontend.php Line: 80 Function: loadpage File: /var/www/html/index.php Line: 334 Function: require_once

Any idea?

joaquinvanschoren commented 6 years ago

Strange. I'm not seeing this on our test server (or my localhost). Does your database contain the task_type table, and it the database connected correctly?

On Thu, Jun 22, 2017 at 7:30 AM Haps notifications@github.com wrote:

After fixing few issues with system configuration, now I am able to select "new data" from menu and I am getting the following error message?

A PHP Error was encountered Severity: Notice

Message: Undefined property: stdClass::$ttid Filename: new/pre.php Line Number: 20 Backtrace: File: /var/www/html/openml_OS/views/pages/frontend/new/pre.php Line: 20 Function: _error_handler

File: /var/www/html/openml_OS/helpers/cms_helper.php Line: 19 Function: view

File: /var/www/html/openml_OS/controllers/Frontend.php Line: 80 Function: loadpage

File: /var/www/html/index.php Line: 334 Function: require_once

Any idea?

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/openml/website/issues/143#issuecomment-310279706, or mute the thread https://github.com/notifications/unsubscribe-auth/ABpQV3Rm08_OrcBWi95ddjE5EaIG-MeZks5sGfwJgaJpZM4NlR95 .

-- Thank you, Joaquin

joaquinvanschoren commented 6 years ago

From @haps-basset on June 22, 2017 22:47

Many thanks for prompt response.

1) Database structure

MariaDB [openml_expdb]> MariaDB [openml_expdb]> show tables; +---------------------------+ | Tables_in_openml_expdb | +---------------------------+ | algorithm | | algorithm_quality |

| setup_tag | | study | | study_tag | | task | | task_inputs | | task_io_types | | task_tag | | task_type | | task_type_inout | | trace | +---------------------------+

MariaDB [openml_expdb]> MariaDB [openml_expdb]> describe task_type;

+--------------+--------------+------+-----+---------+----------------+ | Field | Type | Null | Key | Default | Extra | +--------------+--------------+------+-----+---------+----------------+ | ttid | int(10) | NO | PRI | NULL | auto_increment | | name | varchar(128) | NO | | NULL | | | description | text | NO | | NULL | | | creator | varchar(128) | NO | | NULL | | | contributors | text | YES | | NULL | | | creationDate | datetime | NO | | NULL | | +--------------+--------------+------+-----+---------+----------------+

2) Database connectivity

define( 'DB_NAME_EXPDB', 'openml_expdb' ); define( 'DB_HOST_EXPDB', 'localhost' ); define( 'DB_USER_EXPDB_READ', 'root' ); define( 'DB_PASS_EXPDB_READ', 'xxxxx' ); define( 'DB_USER_EXPDB_WRITE', 'root' ); define( 'DB_PASS_EXPDB_WRITE', 'xxxxxx' );

define( 'DB_NAME_OPENML', 'openml' ); define( 'DB_HOST_OPENML', 'localhost' ); define( 'DB_USER_OPENML', 'root' ); define( 'DB_PASS_OPENML', 'xxxxx' );

I was able to add a new "user" from app's therefore I am assuming connectivity between app and DB is enabled.

MariaDB [openml]> select * from users where id =1; +----+------------+-------------------+----------------------------------+------+-------------------+------------------------------------------+-------------------------+-------------------------+---------------+------------+------------+--------+------------+-----------+-------------+---------+-------+-----------------+-------------+ | id | ip_address | username | password | salt | email | activation_code | forgotten_password_code | forgotten_password_time | remember_code | created_on | last_login | active | first_name | last_name | affiliation | country | image | external_source | external_id | +----+------------+-------------------+----------------------------------+------+-------------------+------------------------------------------+-------------------------+-------------------------+---------------+------------+------------+--------+------------+-----------+-------------+---------+-------+-----------------+-------------+ | 1 | m | root@www.test.org | a29c1864dc52dd6628c622b8dd3b78f2 | NULL | root@www.test.org | 9edc1bd1f4eade7b4187ed133108026cc8e4d848 | NULL | NULL | NULL | 1498106649 | 1498111091 | 1 | root | root | Root | AU | | NULL | NULL | +----+------------+-------------------+----------------------------------+------+-------------------+------------------------------------------+-------------------------+-------------------------+---------------+------------+------------+--------+------------+-----------+-------------+---------+-------+-----------------+-------------+

joaquinvanschoren commented 6 years ago

Users are stored in the OpenML database, while everything else is stored in the expdb database. Maybe the second is not set up correctly? On Fri, 23 Jun 2017 at 00:47, Haps notifications@github.com wrote:

Many thanks for prompt response.

  1. Database structure

MariaDB [openml_expdb]> MariaDB [openml_expdb]> show tables; +---------------------------+ | Tables_in_openml_expdb | +---------------------------+ | algorithm | | algorithm_quality |

| setup_tag | | study | | study_tag | | task | | task_inputs | | task_io_types | | task_tag | | task_type | | task_type_inout | | trace | +---------------------------+

MariaDB [openml_expdb]> MariaDB [openml_expdb]> describe task_type;

+--------------+--------------+------+-----+---------+----------------+ | Field | Type | Null | Key | Default | Extra | +--------------+--------------+------+-----+---------+----------------+ | ttid | int(10) | NO | PRI | NULL | auto_increment | | name | varchar(128) | NO | | NULL | | | description | text | NO | | NULL | | | creator | varchar(128) | NO | | NULL | | | contributors | text | YES | | NULL | | | creationDate | datetime | NO | | NULL | | +--------------+--------------+------+-----+---------+----------------+

  1. Database connectivity

define( 'DB_NAME_EXPDB', 'openml_expdb' ); define( 'DB_HOST_EXPDB', 'localhost' ); define( 'DB_USER_EXPDB_READ', 'root' ); define( 'DB_PASS_EXPDB_READ', 'xxxxx' ); define( 'DB_USER_EXPDB_WRITE', 'root' ); define( 'DB_PASS_EXPDB_WRITE', 'xxxxxx' );

define( 'DB_NAME_OPENML', 'openml' ); define( 'DB_HOST_OPENML', 'localhost' ); define( 'DB_USER_OPENML', 'root' ); define( 'DB_PASS_OPENML', 'xxxxx' );

I was able to add a new "user" from app's therefore I am assuming connectivity between app and DB is enabled.

MariaDB [openml]> select * from users where id =1;

+----+------------+-------------------+----------------------------------+------+-------------------+------------------------------------------+-------------------------+-------------------------+---------------+------------+------------+--------+------------+-----------+-------------+---------+-------+-----------------+-------------+ | id | ip_address | username | password | salt | email | activation_code | forgotten_password_code | forgotten_password_time | remember_code | created_on | last_login | active | first_name | last_name | affiliation | country | image | external_source | external_id |

+----+------------+-------------------+----------------------------------+------+-------------------+------------------------------------------+-------------------------+-------------------------+---------------+------------+------------+--------+------------+-----------+-------------+---------+-------+-----------------+-------------+ | 1 | ��m | root@www.test.org | a29c1864dc52dd6628c622b8dd3b78f2 | NULL | root@www.test.org | 9edc1bd1f4eade7b4187ed133108026cc8e4d848 | NULL | NULL | NULL | 1498106649 | 1498111091 | 1 | root | root | Root | AU | | NULL | NULL |

+----+------------+-------------------+----------------------------------+------+-------------------+------------------------------------------+-------------------------+-------------------------+---------------+------------+------------+--------+------------+-----------+-------------+---------+-------+-----------------+-------------+

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/openml/website/issues/143#issuecomment-310523293, or mute the thread https://github.com/notifications/unsubscribe-auth/ABpQV7FgjSQn9NdMWgzz0EkfJumKG3PGks5sGu73gaJpZM4NlR95 .

-- Thank you, Joaquin

joaquinvanschoren commented 6 years ago

From @haps-basset on June 23, 2017 1:14

We have noticed that the reference to session_hash (column?) in controllers/Api_new.php points to database but none of databases have got any column called session_hash so is it possible that openml.sql schema downloaded form your side is not updated?

Haps

joaquinvanschoren commented 6 years ago

Sorry. This is indeed a recent change. I updated the sql file for the private DB, can you download and import it again?

On Fri, Jun 23, 2017 at 3:14 AM Haps notifications@github.com wrote:

We have noticed that the reference to session_hash (column?) in controllers/Api_new.php points to database but none of databases have got any column called session_hash so is it possible that openml.sql schema downloaded form your side is not updated?

Haps

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/openml/website/issues/143#issuecomment-310543733, or mute the thread https://github.com/notifications/unsubscribe-auth/ABpQV9XqInWkT49-QSePRWikstxLku-0ks5sGxFzgaJpZM4NlR95 .

-- Thank you, Joaquin

joaquinvanschoren commented 6 years ago

From @haps-basset on June 24, 2017 11:52

Are you sure? I am trying to re-create db using a new sql script and I am getting the following error. ERROR 1046 (3D000) at line 22: No database selected

joaquinvanschoren commented 6 years ago

What are you doing exactly when you get this error? Did you just import it on the command line. I usually see this error when running a query without selecting a DB first.

On Sat, Jun 24, 2017 at 1:52 PM Haps notifications@github.com wrote:

Are you sure? I am trying to re-create db using a new sql script and I am getting the following error. ERROR 1046 (3D000) at line 22: No database selected

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/openml/website/issues/143#issuecomment-310834164, or mute the thread https://github.com/notifications/unsubscribe-auth/ABpQV-UXVUvut4ud5DbZy-8EJXkJnxqlks5sHPhygaJpZM4NlR95 .

-- Thank you, Joaquin

joaquinvanschoren commented 6 years ago

From @haps-basset on June 25, 2017 2:21

Yes, you right but I expected script creating database and database structure as it was before not just DB backup. I could re-create database but then how to grant credential to user all, partial etc. which is not described. The maintenance and installation instructions are misleading and laconic. I don’t want to introduce another unknown potential error so that is a reason why I am requesting your support. I want to get this application to work and just forget it. Mysql -u root -p < openml.sql , bear in mind, I dropped previous database instance created by wrong sql script.

joaquinvanschoren commented 6 years ago

From @haps-basset on June 25, 2017 3:31

Well, I have re-created the openml database and restored the content from openml file upgraded by you but outcome is as before. Any attempt to summit a new arff file gives the following error:

A PHP Error was encountered Severity: Notice Message: Undefined property: stdClass::$ttid Filename: new/pre.php Line Number: 20 Backtrace: File: /var/www/html/openml_OS/views/pages/frontend/new/pre.php Line: 20 Function: _error_handler File: /var/www/html/openml_OS/helpers/cms_helper.php Line: 19 Function: view File: /var/www/html/openml_OS/controllers/Frontend.php Line: 80 Function: loadpage File: /var/www/html/index.php Line: 334 Function: require_once describe users

The table users has got missing column see dump +-------------------------+-----------------------+------+-----+-------------------+----------------+ | Field | Type | Null | Key | Default | Extra | +-------------------------+-----------------------+------+-----+-------------------+----------------+ | id | mediumint(8) unsigned | NO | PRI | NULL | auto_increment | | ip_address | varbinary(16) | NO | | NULL | | | username | varchar(100) | NO | | NULL | | | password | varchar(80) | NO | | NULL | | | salt | varchar(40) | YES | | NULL | | | email | varchar(100) | NO | | NULL | | | activation_code | varchar(40) | YES | | NULL | | | forgotten_password_code | varchar(40) | YES | | NULL | | | forgotten_password_time | int(11) unsigned | YES | | NULL | | | remember_code | varchar(40) | YES | | NULL | | | created_on | int(11) unsigned | NO | | NULL | | | last_login | int(11) unsigned | YES | | NULL | | | active | tinyint(1) unsigned | YES | | NULL | | | first_name | varchar(50) | YES | | NULL | | | last_name | varchar(50) | YES | | NULL | | | affiliation | varchar(50) | NO | | NULL | | | country | varchar(50) | NO | | NULL | | | image | varchar(128) | YES | | NULL | | | bio | text | NO | | NULL | | | core | enum('true','false') | NO | | false | | | external_source | varchar(50) | YES | | NULL | | | external_id | varchar(50) | YES | | NULL | | | session_hash | varchar(40) | YES | | NULL | | | session_hash_date | timestamp | YES | | CURRENT_TIMESTAMP | | | gamification_visibility | varchar(32) | NO | | show | | +-------------------------+-----------------------+------+-----+-------------------+----------------+ 25 rows in set (0.07 sec) So what else is missing?

joaquinvanschoren commented 6 years ago

From @haps-basset on June 27, 2017 7:22

Again, hacking php scripts we noticed that the group and users_groups tables are not updated giving us Error 104: This is a read-only account, it does not have permission for write operations.

After updating content of those tables we got Error 132: Failed to move the files

Setting data/ mode to 777 gives us Thanks! Data was uploaded successfully (ID = 1) You can now follow your dataset on OpenML, complete its description, track its impact, create OpenML tasks, and see all ensuing results. You can also continue to add datasets below.

and arff file is moved to /data/dataset/api from /tmp

Unfortunately, data pages shows zero results thus what else is wrong?

The last attempt is just a set of ugly hacks used to find configuration issues. Any idea what is wrong?

joaquinvanschoren commented 6 years ago

Also, if the data pages are empty it's something with elasticsearch. Maybe authentication. Will check tonight. On Tue, 27 Jun 2017 at 09:22, Haps notifications@github.com wrote:

Again, hacking php scripts we noticed that the group and users_groups tables are not updated giving us Error 104: This is a read-only account, it does not have permission for write operations.

After updating content of those tables we got Error 132: Failed to move the files

Setting data/ mode to 777 gives us Thanks! Data was uploaded successfully (ID = 1) You can now follow your dataset on OpenML, complete its description, track its impact, create OpenML tasks, and see all ensuing results. You can also continue to add datasets below.

and arff file is moved to /data/dataset/api from /tmp

Unfortunately, data pages shows zero results thus what else is wrong?

The last attempt is just a set of ugly hacks used to find configuration issues. Any idea what is wrong?

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/openml/website/issues/143#issuecomment-311275026, or mute the thread https://github.com/notifications/unsubscribe-auth/ABpQV31TCwGmXl_zCWE2PDZCuaCLYHtxks5sIK2mgaJpZM4NlR95 .

-- Thank you, Joaquin

joaquinvanschoren commented 6 years ago

From @haps-basset on July 4, 2017 4:8

Referring to your comments about elasticsearch service 1) We are able to create an object and search outside openml ecosystem so I would say that elasticsearch works for us. I am not elasticsearch expert but following available tutorials, I was able to create objects and find it thus I would say that the service works fine. In addition, openml elasticsearch ping gives us a message “All is well” so I would say elasticsearch/kibana and logstash work fine 2) Authentication, I am not sure what is about. Is it related to elastic search installation or in general? We are able to create a new user and activate user’s account. In theory user should be able to add new artifacts but those new items are not registered with elastic search. The elastic search engine was installed without any acl therefore any request is processed (see sniped of BASE_CONFIG-BLANK.php) define( 'ES_URL', 'FILL_IN' ); define( 'ES_USERNAME', 'FILL_IN' ); define( 'ES_PASSWORD', 'FILL_IN' );

joaquinvanschoren commented 6 years ago

Are you still facing issues here? Also wanted to make you aware of this project that provides a Docker image for OpenML: https://github.com/nodechef/openML_docker_server

joaquinvanschoren commented 6 years ago

From @haps-basset on July 22, 2017 9:59

I gave up it is just pointless. Docker solution proposed by you has got this the same problem cannot be integrated with elastic search engine. We have wasted so much time and it is time to stop further investigations.

Haps

From: Joaquin Vanschoren notifications@github.com Sent: Saturday, 22 July 2017 8:58 AM To: openml/website Cc: Haps; Mention Subject: Re: [openml/website] Unable to install OpenML locally (#143)

Are you still facing issues here? Also wanted to make you aware of this project that provides a Docker image for OpenML: https://github.com/nodechef/openML_docker_server

[https://avatars0.githubusercontent.com/u/17507729?v=4&s=400]https://github.com/nodechef/openML_docker_server

nodechef/openML_docker_serverhttps://github.com/nodechef/openML_docker_server github.com openML_docker_server - Server for OPENML website

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/openml/website/issues/143#issuecomment-317134658, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AT6xLOHuVjq4oZ7JMZbtLyL6GjwZeYcAks5sQTQugaJpZM4NlR95.

joaquinvanschoren commented 6 years ago

OK. Will continue this for future users then and add the necessary documentation.

On Sat, Jul 22, 2017 at 11:59 AM Haps notifications@github.com wrote:

I gave up it is just pointless. Docker solution proposed by you has got this the same problem cannot be integrated with elastic search engine. We have wasted so much time and it is time to stop further investigations.

Haps

From: Joaquin Vanschoren notifications@github.com Sent: Saturday, 22 July 2017 8:58 AM To: openml/website Cc: Haps; Mention Subject: Re: [openml/website] Unable to install OpenML locally (#143)

Are you still facing issues here? Also wanted to make you aware of this project that provides a Docker image for OpenML: https://github.com/nodechef/openML_docker_server

[https://avatars0.githubusercontent.com/u/17507729?v=4&s=400]< https://github.com/nodechef/openML_docker_server>

nodechef/openML_docker_server< https://github.com/nodechef/openML_docker_server> github.com openML_docker_server - Server for OPENML website

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub< https://github.com/openml/website/issues/143#issuecomment-317134658>, or mute the thread< https://github.com/notifications/unsubscribe-auth/AT6xLOHuVjq4oZ7JMZbtLyL6GjwZeYcAks5sQTQugaJpZM4NlR95

.

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/openml/website/issues/143#issuecomment-317172405, or mute the thread https://github.com/notifications/unsubscribe-auth/ABpQV559Re1O9nIaje2Bu7Uy_GAU73Q5ks5sQcgOgaJpZM4NlR95 .

-- Thank you, Joaquin

rquintino commented 6 years ago

Hi there, trying to get this running locally (using the docker image), server seems to be up & running, I can register a user and see it on the openml db users table. Openml_expdb was created properly.

But after register a user I get a blank page (http status 200). tried login with that user & valid/invalid passwords, same thing, http 200 blank page. :(

any ideias? public view seems to be ok, (although no data)

didnt noticed errors due to elasticsearch, but didnt configure anything related to that.

I add few things to supplied docker image scripts:

1-(got the latest schema scripts and add both below, original only had openml.sql?)

FROM mysql MAINTAINER Sami

ADD ./latest/openml.sql /docker-entrypoint-initdb.d ADD ./latest/openml_expdb.sql /docker-entrypoint-initdb.d

Also noticed that new on openml/OpenML, there's no index.php, as in the deprecated website repro? am I missing something? (because I tried to run with the latest version from openml/OpenML repro, but complained on the missing index.php.

rquintino commented 6 years ago

ps-on linux had to rename DockerFile to Dockerfile after the git clone (case sensitive)

joaquinvanschoren commented 6 years ago

Sorry, that index.php was dropped when we moved repo's. You can use the one from openml/website or the openml/openml develop branch. Will complete a PR for this soon.

There are indeed some issues with the Docker image. We're building a new one.

rquintino commented 6 years ago

Hi Joaquin, thanks for reaching!

used the develop branch on the docker website mount, and that seemed to fix the issue with login&register (empty page, http 200).

Now was able to login properly (although as used a fake email, account didnt activate, so just forced update openml.users set active=1 to login).

but after login when I go to create new dataset (http://localhost:3000/new/data), I get the same error as Haps above I think.

question: noticed that I don't have any rows in task_type (probably all other tables too), should some data (beyond schema) be initialized in openml_expdb? Any default init/insert script?

A PHP Error was encountered

Severity: Notice

Message: Trying to get property of non-object

Filename: new/pre.php

Line Number: 18

Backtrace:

File: /var/www/html/openml_OS/views/pages/frontend/new/pre.php Line: 18 Function: _error_handler

File: /var/www/html/openml_OS/helpers/cms_helper.php Line: 19 Function: view

File: /var/www/html/openml_OS/controllers/Frontend.php Line: 80 Function: loadpage

File: /var/www/html/index.php Line: 334 Function: require_once

A PHP Error was encountered

Severity: Warning

Message: Creating default object from empty value

Filename: new/pre.php

Line Number: 19

Backtrace:

File: /var/www/html/openml_OS/views/pages/frontend/new/pre.php Line: 19 Function: _error_handler

File: /var/www/html/openml_OS/helpers/cms_helper.php Line: 19 Function: view

File: /var/www/html/openml_OS/controllers/Frontend.php Line: 80 Function: loadpage

File: /var/www/html/index.php Line: 334 Function: require_once

A PHP Error was encountered

Severity: Notice

Message: Undefined property: stdClass::$ttid

Filename: new/pre.php

Line Number: 20

Backtrace:

File: /var/www/html/openml_OS/views/pages/frontend/new/pre.php Line: 20 Function: _error_handler

File: /var/www/html/openml_OS/helpers/cms_helper.php Line: 19 Function: view

File: /var/www/html/openml_OS/controllers/Frontend.php Line: 80 Function: loadpage

File: /var/www/html/index.php Line: 334 Function: require_once

rquintino commented 6 years ago

ok, the init sql scripts seem to be here, /OpenML/blob/develop/data/sql/* ex: /OpenML/blob/develop/data/sql/task_type.sql

what is the proper way of running this in the db?

rquintino commented 6 years ago

ok, one step further, new error run all the .sql files using cat *.sql | mysql -pxxxxx -Dopenml_expdb but the very first Files.sql complained that file table did not exist, skipped that one, all other have run ok

so now, the new data page appears ok, but on submit new dataset I always get an error. like "Could not upload data"

my suspicious: shouldnt there be a file table in the openml_expdb database? FIle.sql seems to init a few datasets, but I could not find the create table code for this table?

ex: INSERT INTO file (id, creator, creation_date, filepath, filesize, filename_original, extension, mime_type, md5_hash, type, access_policy) VALUES (1, 16, '2017-06-18 14:37:03', 'https://www.openml.org/data/download/1666876/phpFsFYVN', 84089, 'anneal.arff', 'arff', 'application/octet-stream', '4eaed8b6ec9d8211024b6c089b064761', 'url', 'public'),

joaquinvanschoren commented 6 years ago

Could it be that you're just missing the private database (which holds the file table)? See here: https://www.openml.org/guide/developers

OpenML uses a second, private database, which holds users and file storage references.

joaquinvanschoren commented 6 years ago

Btw, did you read this? https://github.com/openml/OpenML/wiki/Local-Installation

joaquinvanschoren commented 6 years ago

About the failed test:

027   >>> mice = fetch_openml('miceprotein', version=4, data_home=custom_data_home)
033   >>> mice.data.shape
Expected:
    (1080, 81)
Got:
    (1080, 77)

This happens because that dataset has 4 columns that should NOT be used for modelling (row IDs and features that give away the target). These are likely rightly removed before you return the data, so you probably need to adjust the assertion?

hildeweerts commented 6 years ago

There are indeed some issues with the Docker image. We're building a new one.

@joaquinvanschoren Do you have an update on the status of the new Docker image?

rquintino commented 6 years ago

hi @joaquinvanschoren back to this today

ok, found the issue on the composer/docker, the internal post api call wasnt working (http status 500) because docker-compose it's using 3000 as external port, but as I need to put that in openml base_config/base_url define( 'BASE_URL', 'http://localhost:3000/' ); (otherwise lot of http resources won't load in the browser), then the api tries to connect on the container on the port 3000, which is wrong, and fails. inside the container api port is 80.

Suggestion? don't see a API_URL config option?

https://github.com/nodechef/openML_docker_server/blob/master/docker-compose.yml website: build: ./ ports: - 3000:80 volumes:

rquintino commented 6 years ago

additionaly, tried to create a API_URL config and use that, then API returns that my user is readonly. I created the user through registration form, is there another way of entering the backoffice or something? didn't found any in the docs, thx!

rquintino commented 6 years ago

ps-regarding the file, you were right, wasn't aware that file belongs to private db, thought it was in the public one.

joaquinvanschoren commented 6 years ago

Thanks! @janvanrijn Can we define a port for the API? It doesn't work correctly if the website runs on another port. See above.

joaquinvanschoren commented 6 years ago

If you go to the user profile (log in, click your avatar, click you name) there should be a button 'API authentication' where you can toggle whether your key is read-only or not. It should NOT be read-only by default though. Is this the case for you?