Cant run tests in memory on SQLite3 if fulltextsearch is installed

[robbieaverill]

@extravio commented on Mon Nov 19 2018

When trying to run tests in memory as describe here: https://docs.silverstripe.org/en/3/developer_guides/testing/unit_testing/ I get the following error that originates in SearchUpdater::bind_manipulation_capture()

> sake dev/tests/CssApiTest db=sqlite3 flush=all
ERROR [Warning]: scandir(): Directory name cannot be empty
IN GET /dev/tests/CssApiTest
Line 56 in /var/www/myproject/sqlite3/code/SQLite3SchemaManager.php

Source
======
  47:           $parameters = $this->database->getParameters();
  48:  
  49:           // If in-memory use the current database name only
  50:           if($this->database->getLivesInMemory()) {
  51:                   return array($parameters['database']);
  52:           }
  53:  
  54:           // If using file based database enumerate files in the database directory
  55:           $directory = $parameters['path'];
* 56:           $files = scandir($directory);
  57:  
  58:           // Filter each file in this directory
  59:           $databases = array();
  60:           if($files !== false) foreach($files as $file) {
  61:  
  62:                   // Filter non-files

Trace
=====
scandir()
SQLite3SchemaManager.php:56

SQLite3SchemaManager->databaseList()
SQLite3SchemaManager.php:74

SQLite3SchemaManager->databaseExists(test)
SQLite3Database.php:187

SQLite3Database->selectDatabase(test)
SearchUpdater.php:60

SearchUpdater::bind_manipulation_capture()
SearchUpdater_BindManipulationCaptureFilter.php:8

SearchUpdater_BindManipulationCaptureFilter->preRequest(SS_HTTPRequest,Session,DataModel)
RequestProcessor.php:33

RequestProcessor->preRequest(SS_HTTPRequest,Session,DataModel)
Director.php:142

Director::direct(dev/tests/CssApiTest,DataModel)
cli-script.php:129

PHP Warning:  scandir(): Directory name cannot be empty in /var/www/myproject/sqlite3/code/SQLite3SchemaManager.php on line 56
PHP Stack trace:
PHP   1. {main}() /var/www/myproject/framework/cli-script.php:0
PHP   2. Director::direct() /var/www/myproject/framework/cli-script.php:129
PHP   3. RequestProcessor->preRequest() /var/www/myproject/framework/control/Director.php:142
PHP   4. SearchUpdater_BindManipulationCaptureFilter->preRequest() /var/www/myproject/framework/control/RequestProcessor.php:33
PHP   5. SearchUpdater::bind_manipulation_capture() /var/www/myproject/fulltextsearch/code/search/SearchUpdater_BindManipulationCaptureFilter.php:8
PHP   6. SQLite3Database->selectDatabase() /var/www/myproject/fulltextsearch/code/search/SearchUpdater.php:60
PHP   7. SQLite3SchemaManager->databaseExists() /var/www/myproject/sqlite3/code/SQLite3Database.php:187
PHP   8. SQLite3SchemaManager->databaseList() /var/www/myproject/sqlite3/code/SQLite3SchemaManager.php:74
PHP   9. scandir() /var/www/myproject/sqlite3/code/SQLite3SchemaManager.php:56
ERROR [User Error]: Attempted to connect to non-existing database "test"
IN GET /dev/tests/CssApiTest
Line 191 in /var/www/myproject/sqlite3/code/SQLite3Database.php

Source
======
  182:  public function getDatabaseServer() {
  183:          return "sqlite";
  184:  }
  185: 
  186:  public function selectDatabase($name, $create = false, $errorLevel = E_USER_ERROR) {
  187:          if (!$this->schemaManager->databaseExists($name)) {
  188:                  // Check DB creation permisson
  189:                  if (!$create) {
  190:                          if ($errorLevel !== false) {
* 191:                                  user_error("Attempted to connect to non-existing database \"$name\"", $errorLevel);
  192:                          }
  193:                          // Unselect database
  194:                          $this->connector->unloadDatabase();
  195:                          return false;
  196:                  }
  197:                  $this->schemaManager->createDatabase($name);

Trace
=====
user_error(Attempted to connect to non-existing database "test",256)
SQLite3Database.php:191

SQLite3Database->selectDatabase(test)
SearchUpdater.php:60

SearchUpdater::bind_manipulation_capture()
SearchUpdater_BindManipulationCaptureFilter.php:8

SearchUpdater_BindManipulationCaptureFilter->preRequest(SS_HTTPRequest,Session,DataModel)
RequestProcessor.php:33

RequestProcessor->preRequest(SS_HTTPRequest,Session,DataModel)
Director.php:142

Director::direct(dev/tests/CssApiTest,DataModel)
cli-script.php:12

[NightJar]

@extravio This seems like custom code is the cause of this issue. A critical configuration option is missing, so that part error makes sense (the path cannot be null), which proceeds to become a missing database (presumably because "test.sqlite" !== "some/path/test.sqlite").

You've made an error in calling the sqlite3 adaptor when testing, so I can only assume that you have custom code powering the creation of this test.

The db parameter should be passed via an environment variable, not as a CLI equivocation of a GET parameter in order to trigger framework into using the sqlite adaptor (if it is installed). The variable should also be upper case.

DB=SQLITE vendor/bin/sake dev/tests/CssApiTest

See here: https://github.com/silverstripe/silverstripe-framework/blob/4/tests/bootstrap/environment.php#L19

However this is all assuming you're running SilverStripe 4 with the latest 2.x-dev of this module, which may not be true as it appears you're running sake as opposed to vendor/bin/sake (which may be an alias, so it's not a given).

If you're running SilverStripe 3 (your classes are missing namespaces) then the environment style call relies on the presence of the silverstripe/travis-support module.

As this appears to be custom code I'm afraid you're going to have to give more specifics as to what you're attempting to do and why you think this is a bug with this module as opposed to with your project's code.

In the very least: With SilverStripe 3 you will need to set the path in the database config:

https://github.com/silverstripe/silverstripe-sqlite3/blob/2bde2640c31a62708af31b1fbeffcd91ff5a4e62/_configure_database.php#L8

or define it as an environment variable in SilverStripe 4.

[extravio]

@NightJar Thanks for your reply

As you've noticed, the bug was originally raised under the fulltextsearch module https://github.com/silverstripe/silverstripe-fulltextsearch/issues/238 and @robbieaverill moved it here.

I'm using Silverstripe 3.7 and as described in the section "Use SQLite In Memory" in the docs https://docs.silverstripe.org/en/3/developer_guides/testing/unit_testing/, I added the following to my _config.php

if(Director::isDev()) {
    if(isset($_GET['db']) && ($db = $_GET['db'])) {
        global $databaseConfig;
        if($db == 'sqlite3') {
            $databaseConfig = array(
                                "type" => 'SQLite3Database',
                                "server" => 'none',
                                "username" => 'none',
                                "password" => 'none',
                                "database" => 'test',
                                "path" => ":memory:",
                            );
        }
    }
}

and > sake dev/tests/CssApiTest db=sqlite3 works fine as long as the fulltextsearch module is not installed.

Should this issue be reopened? https://github.com/silverstripe/silverstripe-fulltextsearch/issues/238

[extravio]

Hi @NightJar, You commented the following on the fulltextsearch issue:

this is due to a database proxy that FulltextSearch fits around the actual database connection (sqlite3 in this case). This is how a search index document update is triggered after a write operation.

Running tests on MySQL with the fulltextsearch module installed works fine. What is different when running the tests on Sqlite3? Why is it that the fulltextsearch module does not pick up Sqlite3?

[NightJar]

It will probably be something to do with the way the database proxy setup in the SilverStripe 3 version of FulltextSearch.

In SilverStripe 4 this was abstracted to a more defined manner, but in SS3 it's still very free-from and every module does it differently. I'll have to investigate how FulltextSearch does this to know whether it's an issue there or with the way this module handles it.

But my suspicions are that there's probably some assumptions about the server type backing the connection class in the proxy code.