matiasdelellis / facerecognition

Nextcloud app that implement a basic facial recognition system.
GNU Affero General Public License v3.0
527 stars 47 forks source link

"Doctrine\DBAL\DBALException" Error #127

Closed emresaracoglu closed 5 years ago

emresaracoglu commented 5 years ago

I have a CentOS 7 operating system installed on my current server, and it works very well but I've created a second virtual machine for your application and installed Ubuntu 18.10 - Nextcloud software.

Beacause I getting this error: CentOS I got this error [view] if not I'm quite satisfied with CentOS 👍

I wrote the command to create a face set on the Ubuntu: occ face:background_job -u [username]

Click for Console ``` root@server:/var/www/html# sudo -u www-data php occ face:background_job -u [username] An unhandled exception has been thrown: Doctrine\DBAL\DBALException: Failed to connect to the database: An exception occured in driver: could not find driver in /var/www/html/lib/private/DB/Connection.php:64 Stack trace: #0 /var/www/html/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(429): OC\DB\Connection->connect() #1 /var/www/html/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(389): Doctrine\DBAL\Connection->getDatabasePlatformVersion() #2 /var/www/html/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(328): Doctrine\DBAL\Connection->detectDatabasePlatform() #3 /var/www/html/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(623): Doctrine\DBAL\Connection->getDatabasePlatform() #4 /var/www/html/lib/private/DB/Connection.php(151): Doctrine\DBAL\Connection->setTransactionIsolation(2) #5 /var/www/html/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/DriverManager.php(172): OC\DB\Connection->__construct(Array, Object(Doctrine\DBAL\Driver\PDOMySql\Driver), Object(Doctrine\DBAL\Configuration), Object(Doctrine\Common\EventManager)) #6 /var/www/html/lib/private/DB/ConnectionFactory.php(158): Doctrine\DBAL\DriverManager::getConnection(Array, Object(Doctrine\DBAL\Configuration), Object(Doctrine\Common\EventManager)) #7 /var/www/html/lib/private/Server.php(641): OC\DB\ConnectionFactory->getConnection('mysql', Array) #8 /var/www/html/3rdparty/pimple/pimple/src/Pimple/Container.php(118): OC\Server->OC\{closure}(Object(OC\Server)) #9 /var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php(117): Pimple\Container->offsetGet('OCP\\IDBConnecti...') #10 /var/www/html/lib/private/ServerContainer.php(132): OC\AppFramework\Utility\SimpleContainer->query('OCP\\IDBConnecti...') #11 /var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php(165): OC\ServerContainer->query('OCP\\IDBConnecti...') #12 /var/www/html/3rdparty/pimple/pimple/src/Pimple/Container.php(114): OC\AppFramework\Utility\SimpleContainer->OC\AppFramework\Utility\{closure}(Object(OC\Server)) #13 /var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php(117): Pimple\Container->offsetGet('DatabaseConnect...') #14 /var/www/html/lib/private/ServerContainer.php(132): OC\AppFramework\Utility\SimpleContainer->query('DatabaseConnect...') #15 /var/www/html/lib/private/Server.php(1552): OC\ServerContainer->query('DatabaseConnect...') #16 /var/www/html/lib/private/Server.php(360): OC\Server->getDatabaseConnection() #17 /var/www/html/3rdparty/pimple/pimple/src/Pimple/Container.php(118): OC\Server->OC\{closure}(Object(OC\Server)) #18 /var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php(117): Pimple\Container->offsetGet('OC\\Authenticati...') #19 /var/www/html/lib/private/ServerContainer.php(132): OC\AppFramework\Utility\SimpleContainer->query('OC\\Authenticati...') #20 /var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php(68): OC\ServerContainer->query('OC\\Authenticati...') #21 /var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php(98): OC\AppFramework\Utility\SimpleContainer->buildClass(Object(ReflectionClass)) #22 /var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php(119): OC\AppFramework\Utility\SimpleContainer->resolve('OC\\Authenticati...') #23 /var/www/html/lib/private/ServerContainer.php(132): OC\AppFramework\Utility\SimpleContainer->query('OC\\Authenticati...') #24 /var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php(68): OC\ServerContainer->query('OC\\Authenticati...') #25 /var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php(98): OC\AppFramework\Utility\SimpleContainer->buildClass(Object(ReflectionClass)) #26 /var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php(119): OC\AppFramework\Utility\SimpleContainer->resolve('OC\\Authenticati...') #27 /var/www/html/lib/private/ServerContainer.php(132): OC\AppFramework\Utility\SimpleContainer->query('OC\\Authenticati...') #28 /var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php(165): OC\ServerContainer->query('OC\\Authenticati...') #29 /var/www/html/3rdparty/pimple/pimple/src/Pimple/Container.php(114): OC\AppFramework\Utility\SimpleContainer->OC\AppFramework\Utility\{closure}(Object(OC\Server)) #30 /var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php(117): Pimple\Container->offsetGet('OC\\Authenticati...') #31 /var/www/html/lib/private/ServerContainer.php(132): OC\AppFramework\Utility\SimpleContainer->query('OC\\Authenticati...') #32 /var/www/html/lib/private/Server.php(372): OC\ServerContainer->query('OC\\Authenticati...') #33 /var/www/html/3rdparty/pimple/pimple/src/Pimple/Container.php(118): OC\Server->OC\{closure}(Object(OC\Server)) #34 /var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php(117): Pimple\Container->offsetGet('OC\\User\\Session') #35 /var/www/html/lib/private/ServerContainer.php(132): OC\AppFramework\Utility\SimpleContainer->query('OC\\User\\Session') #36 /var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php(165): OC\ServerContainer->query('OC\\User\\Session') #37 /var/www/html/3rdparty/pimple/pimple/src/Pimple/Container.php(114): OC\AppFramework\Utility\SimpleContainer->OC\AppFramework\Utility\{closure}(Object(OC\Server)) #38 /var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php(117): Pimple\Container->offsetGet('UserSession') #39 /var/www/html/lib/private/ServerContainer.php(132): OC\AppFramework\Utility\SimpleContainer->query('UserSession') #40 /var/www/html/lib/private/Server.php(1428): OC\ServerContainer->query('UserSession') #41 /var/www/html/lib/private/Server.php(694): OC\Server->getUserSession() #42 /var/www/html/3rdparty/pimple/pimple/src/Pimple/Container.php(118): OC\Server->OC\{closure}(Object(OC\Server)) #43 /var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php(117): Pimple\Container->offsetGet('OC\\App\\AppManag...') #44 /var/www/html/lib/private/ServerContainer.php(132): OC\AppFramework\Utility\SimpleContainer->query('OC\\App\\AppManag...') #45 /var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php(165): OC\ServerContainer->query('OC\\App\\AppManag...') #46 /var/www/html/3rdparty/pimple/pimple/src/Pimple/Container.php(114): OC\AppFramework\Utility\SimpleContainer->OC\AppFramework\Utility\{closure}(Object(OC\Server)) #47 /var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php(117): Pimple\Container->offsetGet('AppManager') #48 /var/www/html/lib/private/ServerContainer.php(132): OC\AppFramework\Utility\SimpleContainer->query('AppManager') #49 /var/www/html/lib/private/Server.php(1723): OC\ServerContainer->query('AppManager') #50 /var/www/html/lib/private/legacy/app.php(342): OC\Server->getAppManager() #51 /var/www/html/lib/private/legacy/app.php(113): OC_App::getEnabledApps() #52 /var/www/html/lib/base.php(654): OC_App::loadApps(Array) #53 /var/www/html/lib/base.php(1068): OC::init() #54 /var/www/html/console.php(46): require_once('/var/www/html/l...') #55 /var/www/html/occ(11): require_once('/var/www/html/c...') #56 {main}PHP Fatal error: Uncaught Doctrine\DBAL\DBALException: Failed to connect to the database: An exception occured in driver: could not find driver in /var/www/html/lib/private/DB/Connection.php:64 Stack trace: #0 /var/www/html/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(429): OC\DB\Connection->connect() #1 /var/www/html/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(389): Doctrine\DBAL\Connection->getDatabasePlatformVersion() #2 /var/www/html/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(328): Doctrine\DBAL\Connection->detectDatabasePlatform() #3 /var/www/html/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(623): Doctrine\DBAL\Connection->getDatabasePlatform() #4 /var/www/html/lib/private/DB/Connection.php(151): Doctrine\DBAL\Connection->setTransactionIsolation(2) #5 /var/www/html/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/DriverManager.php(172): OC\DB\Connection->__construct(Array, Object(Doctrine\DBAL\Driver\PDOMySql\Driver), Object(Doctrine\DBAL\Configuration), Object(Doctrine\Common\Event in /var/www/html/lib/private/DB/Connection.php on line 64 ```
stalker314314 commented 5 years ago

So, let me see if I understand - in your VM you have Ubuntu on which there is Nextcloud installation and which has FaceRecognition app? I think this has nothing to do with FaceRecognition, seems as you cannot connect you your database at all? Does Nextcloud works at all in that Ubuntu VM? I think you would get same error if you execute sudo -u www-data php occ user:list.

Basically, you want FaceRecognition app on same machine and same Nextcloud installation as your main one. That means you either:

Regarding your original issue, let me install CentOS in VM and try to compile it there and see if we can help with fix root cause.

emresaracoglu commented 5 years ago

I don't want to interfere with CentOS (At least today 😄) Maybe I can deal with CentOS tomorrow. Thank you for your help.


I erased the virtual machine I was talking about (Ubuntu 18.10) and I installed fresh Ubuntu on a new LXC but this time with this:

sudo bash -c "$(wget -q -O - https://raw.githubusercontent.com/nextcloud/vm/master/nextcloud_install_production.sh)"

root@Datarals:~/pdlib# make
/bin/bash /root/pdlib/libtool --mode=compile g++ -DZEND_ENABLE_STATIC_TSRMLS_CACHE=1 -I. -I/root/pdlib -DPHP_ATOM_INC -I/root/pdlib/include -I/root/pdlib/main -I/root/pdlib -I/usr/include/php/20170718 -I/usr/include/php/20170718/main -I/usr/include/php/20170718/TSRM -I/usr/include/php/20170718/Zend -I/usr/include/php/20170718/ext -I/usr/include/php/20170718/ext/date/lib -I/usr/local/include  -DHAVE_CONFIG_H  -std=c++11   -c /root/pdlib/src/face_detection.cc -o src/face_detection.lo
libtool: compile:  g++ -DZEND_ENABLE_STATIC_TSRMLS_CACHE=1 -I. -I/root/pdlib -DPHP_ATOM_INC -I/root/pdlib/include -I/root/pdlib/main -I/root/pdlib -I/usr/include/php/20170718 -I/usr/include/php/20170718/main -I/usr/include/php/20170718/TSRM -I/usr/include/php/20170718/Zend -I/usr/include/php/20170718/ext -I/usr/include/php/20170718/ext/date/lib -I/usr/local/include -DHAVE_CONFIG_H -std=c++11 -c /root/pdlib/src/face_detection.cc  -fPIC -DPIC -o src/.libs/face_detection.o
In file included from /usr/local/include/dlib/gui_core/xlib.h:4:0,
                 from /usr/local/include/dlib/gui_core.h:14,
                 from /usr/local/include/dlib/gui_widgets/drawable.h:10,
                 from /usr/local/include/dlib/gui_widgets/widgets.h:16,
                 from /usr/local/include/dlib/gui_widgets.h:13,
                 from /root/pdlib/src/face_detection.cc:6:
/usr/local/include/dlib/gui_core/gui_core_kernel_2.h:11:2: error: #error "DLIB_NO_GUI_SUPPORT is defined so you can't use the GUI code.  Turn DLIB_NO_GUI_SUPPORT off if you want to use it."
 #error "DLIB_NO_GUI_SUPPORT is defined so you can't use the GUI code.  Turn DLIB_NO_GUI_SUPPORT off if you want to use it."
  ^~~~~
/usr/local/include/dlib/gui_core/gui_core_kernel_2.h:12:2: error: #error "Also make sure you have libx11-dev installed on your system"
 #error "Also make sure you have libx11-dev installed on your system"
  ^~~~~
Makefile:198: recipe for target 'src/face_detection.lo' failed
make: *** [src/face_detection.lo] Error 1

I tried;

root@Datarals:~# sudo apt-get install libx11-dev
Reading package lists... Done
Building dependency tree
Reading state information... Done
libx11-dev is already the newest version (2:1.6.4-3ubuntu0.2).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

Do you have any idea about this?

stalker314314 commented 5 years ago

Did you take a look at https://github.com/goodspb/pdlib/issues/12? Seems like Slavik already hit this issue. Basically, you need libx11-dev before installing DLib. So, install that one, and remove dlib/build directory (which you used to compile dlib) and recompile dlib. Hope this will help!:)

BTW, since you are doing this in VM, I hope you have enough memory? Better to tell you now than you find out later:) Check Requirements and limitations

emresaracoglu commented 5 years ago

Result :)

root@Datarals:/var/www/nextcloud# sudo -u www-data php occ face:background_job -u emresaracoglu
1/9 - Executing task CheckRequirementsTask (Check all requirements)
        PDLib is not loaded. Cannot continue
Task CheckRequirementsTask signalled we should not continue, bailing out

Module Path:

/root/pdlib/modules/pdlib.so (22MB~)

I added in PHP.ini files:

/etc/php/7.2/apache2/php.ini & /etc/php/7.2/fpm/php.ini

[pdlib]
extension="pdlib.so"

I guess not. I can not do this. If you have another idea, I'll keep going, otherwise it's enough for me :)


Yes. I'm not having any trouble with memory, I've created memory 32 GB for Ubuntu

emresaracoglu commented 5 years ago

I'm going back to the snapshot image. I will try again with a clean installation.

stalker314314 commented 5 years ago

I think you are almost there:) I think you also need to add pdlib.so in /etc/php/cli/php.ini (same what you already did, but in "cli", as you are using php from command line). This is why you get that pdlib is not installed

emresaracoglu commented 5 years ago

I've included it in the cli file 👍 and I've successfully installed! This is the hardest plugin I've ever installation 😄

I made a few face scanning in a few photos and face detected but

Your friends have not been recognized yet Please, be patient

I'm still getting the this text. What is the reason of this?

Console:

root@Datarals:/var/www/nextcloud# sudo -u www-data php occ face:background_job -u emresaracoglu
1/9 - Executing task CheckRequirementsTask (Check all requirements)
2/9 - Executing task CheckCronTask (Check that service is started from either cron or from command)
3/9 - Executing task LockTask (Acquire lock so that only one background task can run)
4/9 - Executing task StaleImagesRemovalTask (Crawl for stale images (either missing in filesystem or under .nomedia) and remove them from DB)
5/9 - Executing task CreateClustersTask (Create new persons or update existing persons)
        Skipping cluster creation, not enough data (yet) collected. For cluster creation, you need either one of the following:
        * have 1000 faces already processed (you have 1),
        * have 100 images (you have 13),
        * or you need to have 95% of you images processed (you have 0.38%)
6/9 - Executing task AddMissingImagesTask (Crawl for missing images for each user and insert them in DB)
7/9 - Executing task EnumerateImagesMissingFacesTask (Find all images which don't have faces generated for them)
8/9 - Executing task ImageProcessingTask (Process all images to extract faces)
        NOTE: Starting face recognition. If you experience random crashes after this point, please look FAQ at https://github.com/matiasdelellis/facerecognition/wiki/FAQ
        Processing image /mnt/ncdata/emresaracoglu/files/IMG_20180827_211740.jpg
        Faces found: 1
        Processing image /mnt/ncdata/emresaracoglu/files/IMG_20180827_215506.jpg
        Faces found: 0
        File with ID 178 doesn't exist anymore, skipping it
        Processing image /mnt/ncdata/emresaracoglu/files/IMG_20180827_184732.jpg
        Faces found: 2
        File with ID 181 doesn't exist anymore, skipping it
        File with ID 179 doesn't exist anymore, skipping it
        Processing image /mnt/ncdata/emresaracoglu/files/IMG_20180827_233421.jpg
        Faces found: 1
        Processing image /mnt/ncdata/emresaracoglu/files/IMG_20180827_215437.jpg
        Faces found: 1
9/9 - Executing task UnlockTask (Release obtained lock)
You have new mail in /var/mail/root
stalker314314 commented 5 years ago

So, whole idea with face:background_job is to run it continuously, over and over again. Usage section is still empty:), and we need to be better at explaining this to users. So, you put it to cron, or run it from time to time. For example, if you run same command again, you will get output, telling you why there is no clusters yet (or maybe it will be created when you run it again?) If you get in output something like this line, then you don't have enough condition satisfied.


I know it is hard, right?:) Hopefully, all this fiddling with compling dlib/pdlib will be gone and replaced with simple sudo apt-get install. And it will be installed from app store in future... It will still not be as easy as some other apps (as you need to use command line to set up cron to run face recognition...)

emresaracoglu commented 5 years ago

I noticed that the photo was not square on the face. Maybe in advanced versions you might be thinking about it :)

Also I did a test,

First step: The application found and marked the faces in the photos and showed on admin panel.

Second Step: After, I've uploaded new photos and fthe application found faces in the new photos but it did not show on the admin panel & details of the photo.

Could this be a bug?

Console:

root@Datarals:/var/www/nextcloud# sudo -u www-data php occ face:background_job -u emresaracoglu -t 4500
1/9 - Executing task CheckRequirementsTask (Check all requirements)
2/9 - Executing task CheckCronTask (Check that service is started from either cron or from command)
3/9 - Executing task LockTask (Acquire lock so that only one background task can run)
4/9 - Executing task StaleImagesRemovalTask (Crawl for stale images (either missing in filesystem or under .nomedia) and remove them from DB)
5/9 - Executing task CreateClustersTask (Create new persons or update existing persons)
        Clusters already exist, estimated there is no need to recreate them
6/9 - Executing task AddMissingImagesTask (Crawl for missing images for each user and insert them in DB)
7/9 - Executing task EnumerateImagesMissingFacesTask (Find all images which don't have faces generated for them)
8/9 - Executing task ImageProcessingTask (Process all images to extract faces)
        NOTE: Starting face recognition. If you experience random crashes after this point, please look FAQ at https://github.com/matiasdelellis/facerecognition/wiki/FAQ
9/9 - Executing task UnlockTask (Release obtained lock)
matiasdelellis commented 5 years ago

Hi @emresaracoglu I'm glad that I'm testing it, and you give us as much feedback as possible.. :smile:

As @stalker314314 commented, It is designed for continuous execution (every 15 minutes?). It can be confusing when you execute it by hand, but in practice it works well.

In short, within the steps, first the grouping is done (Step 5), and then search for new faces (Steps 6, 7 and 8). Therefore, this requires that when you add new photos, in an execution they are analyzed (Find faces, which is the most difficult task), and in the second execution they are grouped.

emresaracoglu commented 5 years ago

@matiasdelellis Application doing a good job with CRON. Scanned all faces and clustered!👍 @stalker314314 Thank you for your help for yesterday

Now I have to think about the big question mark for me. How will the current cloud (CentOS) be moved to the new cloud (Ubuntu)?

Not many members are registered (9 Members). The capacity used is 210 GB~ but the files available are small size files (Docs, PDF, etc.) because of the boredom. I can do RSYNC with local IPs, but can you give an idea about the database - how can MySQL database be converted to Postgres DB?

Update //

I hope my plan works properly.

stalker314314 commented 5 years ago

@emresaracoglu I am not expert on Nextcloud (not sure about @matiasdelellis :)) I cannot give you any advise around this, this is huge operation. One hint though: after you move DB and all files, don't forget to run occ files:scan to rescan all files (and repopulate DB, if needed)

@matiasdelellis This is now 3rd or 4th time that people are complaining about not having clusters after script is done. I had it enough now:) I think it would vastly improve first-time experience if we improve that part. I was thinking to add same task (CreateClustersTask), but at the end (between ImageProcessingTask and UnlockTask). It would not be run always - only if previous task (5/9) didn't find anything. If 5/9 task already did some clusters, this one could be skipped. What do you think, @matiasdelellis?

matiasdelellis commented 5 years ago

Hi @emresaracoglu , The migration of servers is quite documented, but the complicated thing is if you change the database.. I'm not sure how to solve that.

You can try to export it in MySQL and inport it with progress but I'm not sure how it will turn out .. If you have 9 users, I understand that you must necessarily migrate the database, so that they keep the shared files, etc. You can not start from scratch .. :disappointed:

matiasdelellis commented 5 years ago

@matiasdelellis This is now 3rd or 4th time that people are complaining about not having clusters after script is done. I had it enough now:) I think it would vastly improve first-time experience if we improve that part. I was thinking to add same task (CreateClustersTask), but at the end (between ImageProcessingTask and UnlockTask). It would not be run always - only if previous task (5/9) didn't find anything. If 5/9 task already did some clusters, this one could be skipped. What do you think, @matiasdelellis?

@stalker314314 I guess I was one of the first to discuss this .. :sweat_smile:

This order of operation is justified when executed as a scheduled task. We could consider that it is a scheduled task, when executed limited by timeout? If it is executed without timeout, it means that the admin tries to analyze all the photos and the time does not matter. In this case we can not reverse the tasks to analyze first and group at the end?

emresaracoglu commented 5 years ago

@stalker314314 I will use this command if the database fail to restore. @matiasdelellis Thank you for link, very useful.


I realized that the cluster is not working stably. Also, the application recognizes the same person's face as a different face. This is of course natural.

Therefore, I think we should be able to manually merge the faces (app ask to user: "Are these the same person? they will be merged togetger." if when the same name for face is written by user). I believe this feature will be great if you are the think for the future versions.

matiasdelellis commented 5 years ago

I realized that the cluster is not working stably.

The stability of the groups is assured with a greater amount of images... But we try to improve it ..

Also, the application recognizes the same person's face as a different face. This is of course natural.

Unhappily, we must accept some margin of error.. :disappointed:

Therefore, I think we should be able to manually merge the faces (app ask to user: "Are these the same person? they will be merged togetger." if when the same name for face is written by user). I believe this feature will be great if you are the think for the future versions.

This is still in discussion... but for this version we will try to trust that automatic clustering.. and with the feedback of more users we will improve..

stalker314314 commented 5 years ago

I think we should be able to manually merge the faces

Yes, we mentioned that in functional spec, in user operation here, as joining clusters. Not in this version (Google Photos do not have that too:D)

@stalker314314 I guess I was one of the first to discuss this .. sweat_smile

Yes, you are! And you were right. With this amount of evidence + your input, and I changed my mind:) I don't want to be stubborn old man:)

matiasdelellis commented 5 years ago

Hi, I understand that this report was resolved, and I close it. Please, if necessary, open it again. :smile: