Closed emresaracoglu closed 5 years ago
So, let me see if I understand - in your VM you have Ubuntu on which there is Nextcloud installation and which has FaceRecognition app? I think this has nothing to do with FaceRecognition, seems as you cannot connect you your database at all? Does Nextcloud works at all in that Ubuntu VM? I think you would get same error if you execute sudo -u www-data php occ user:list
.
Basically, you want FaceRecognition app on same machine and same Nextcloud installation as your main one. That means you either:
Regarding your original issue, let me install CentOS in VM and try to compile it there and see if we can help with fix root cause.
I don't want to interfere with CentOS (At least today 😄) Maybe I can deal with CentOS tomorrow. Thank you for your help.
I erased the virtual machine I was talking about (Ubuntu 18.10) and I installed fresh Ubuntu on a new LXC but this time with this:
sudo bash -c "$(wget -q -O - https://raw.githubusercontent.com/nextcloud/vm/master/nextcloud_install_production.sh)"
make
command in the Dlib installation process.root@Datarals:~/pdlib# make
/bin/bash /root/pdlib/libtool --mode=compile g++ -DZEND_ENABLE_STATIC_TSRMLS_CACHE=1 -I. -I/root/pdlib -DPHP_ATOM_INC -I/root/pdlib/include -I/root/pdlib/main -I/root/pdlib -I/usr/include/php/20170718 -I/usr/include/php/20170718/main -I/usr/include/php/20170718/TSRM -I/usr/include/php/20170718/Zend -I/usr/include/php/20170718/ext -I/usr/include/php/20170718/ext/date/lib -I/usr/local/include -DHAVE_CONFIG_H -std=c++11 -c /root/pdlib/src/face_detection.cc -o src/face_detection.lo
libtool: compile: g++ -DZEND_ENABLE_STATIC_TSRMLS_CACHE=1 -I. -I/root/pdlib -DPHP_ATOM_INC -I/root/pdlib/include -I/root/pdlib/main -I/root/pdlib -I/usr/include/php/20170718 -I/usr/include/php/20170718/main -I/usr/include/php/20170718/TSRM -I/usr/include/php/20170718/Zend -I/usr/include/php/20170718/ext -I/usr/include/php/20170718/ext/date/lib -I/usr/local/include -DHAVE_CONFIG_H -std=c++11 -c /root/pdlib/src/face_detection.cc -fPIC -DPIC -o src/.libs/face_detection.o
In file included from /usr/local/include/dlib/gui_core/xlib.h:4:0,
from /usr/local/include/dlib/gui_core.h:14,
from /usr/local/include/dlib/gui_widgets/drawable.h:10,
from /usr/local/include/dlib/gui_widgets/widgets.h:16,
from /usr/local/include/dlib/gui_widgets.h:13,
from /root/pdlib/src/face_detection.cc:6:
/usr/local/include/dlib/gui_core/gui_core_kernel_2.h:11:2: error: #error "DLIB_NO_GUI_SUPPORT is defined so you can't use the GUI code. Turn DLIB_NO_GUI_SUPPORT off if you want to use it."
#error "DLIB_NO_GUI_SUPPORT is defined so you can't use the GUI code. Turn DLIB_NO_GUI_SUPPORT off if you want to use it."
^~~~~
/usr/local/include/dlib/gui_core/gui_core_kernel_2.h:12:2: error: #error "Also make sure you have libx11-dev installed on your system"
#error "Also make sure you have libx11-dev installed on your system"
^~~~~
Makefile:198: recipe for target 'src/face_detection.lo' failed
make: *** [src/face_detection.lo] Error 1
I tried;
root@Datarals:~# sudo apt-get install libx11-dev
Reading package lists... Done
Building dependency tree
Reading state information... Done
libx11-dev is already the newest version (2:1.6.4-3ubuntu0.2).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Do you have any idea about this?
Did you take a look at https://github.com/goodspb/pdlib/issues/12? Seems like Slavik already hit this issue. Basically, you need libx11-dev before installing DLib. So, install that one, and remove dlib/build
directory (which you used to compile dlib) and recompile dlib. Hope this will help!:)
BTW, since you are doing this in VM, I hope you have enough memory? Better to tell you now than you find out later:) Check Requirements and limitations
/root
Result :)
root@Datarals:/var/www/nextcloud# sudo -u www-data php occ face:background_job -u emresaracoglu
1/9 - Executing task CheckRequirementsTask (Check all requirements)
PDLib is not loaded. Cannot continue
Task CheckRequirementsTask signalled we should not continue, bailing out
Module Path:
/root/pdlib/modules/pdlib.so
(22MB~)
I added in PHP.ini files:
/etc/php/7.2/apache2/php.ini
& /etc/php/7.2/fpm/php.ini
[pdlib]
extension="pdlib.so"
I guess not. I can not do this. If you have another idea, I'll keep going, otherwise it's enough for me :)
Yes. I'm not having any trouble with memory, I've created memory 32 GB for Ubuntu
I'm going back to the snapshot image. I will try again with a clean installation.
I think you are almost there:) I think you also need to add pdlib.so in /etc/php/cli/php.ini (same what you already did, but in "cli", as you are using php from command line). This is why you get that pdlib is not installed
I've included it in the cli
file 👍 and I've successfully installed!
This is the hardest plugin I've ever installation 😄
I made a few face scanning in a few photos and face detected but
Your friends have not been recognized yet Please, be patient
I'm still getting the this text. What is the reason of this?
Console:
root@Datarals:/var/www/nextcloud# sudo -u www-data php occ face:background_job -u emresaracoglu
1/9 - Executing task CheckRequirementsTask (Check all requirements)
2/9 - Executing task CheckCronTask (Check that service is started from either cron or from command)
3/9 - Executing task LockTask (Acquire lock so that only one background task can run)
4/9 - Executing task StaleImagesRemovalTask (Crawl for stale images (either missing in filesystem or under .nomedia) and remove them from DB)
5/9 - Executing task CreateClustersTask (Create new persons or update existing persons)
Skipping cluster creation, not enough data (yet) collected. For cluster creation, you need either one of the following:
* have 1000 faces already processed (you have 1),
* have 100 images (you have 13),
* or you need to have 95% of you images processed (you have 0.38%)
6/9 - Executing task AddMissingImagesTask (Crawl for missing images for each user and insert them in DB)
7/9 - Executing task EnumerateImagesMissingFacesTask (Find all images which don't have faces generated for them)
8/9 - Executing task ImageProcessingTask (Process all images to extract faces)
NOTE: Starting face recognition. If you experience random crashes after this point, please look FAQ at https://github.com/matiasdelellis/facerecognition/wiki/FAQ
Processing image /mnt/ncdata/emresaracoglu/files/IMG_20180827_211740.jpg
Faces found: 1
Processing image /mnt/ncdata/emresaracoglu/files/IMG_20180827_215506.jpg
Faces found: 0
File with ID 178 doesn't exist anymore, skipping it
Processing image /mnt/ncdata/emresaracoglu/files/IMG_20180827_184732.jpg
Faces found: 2
File with ID 181 doesn't exist anymore, skipping it
File with ID 179 doesn't exist anymore, skipping it
Processing image /mnt/ncdata/emresaracoglu/files/IMG_20180827_233421.jpg
Faces found: 1
Processing image /mnt/ncdata/emresaracoglu/files/IMG_20180827_215437.jpg
Faces found: 1
9/9 - Executing task UnlockTask (Release obtained lock)
You have new mail in /var/mail/root
So, whole idea with face:background_job
is to run it continuously, over and over again. Usage section is still empty:), and we need to be better at explaining this to users. So, you put it to cron, or run it from time to time. For example, if you run same command again, you will get output, telling you why there is no clusters yet (or maybe it will be created when you run it again?) If you get in output something like this line, then you don't have enough condition satisfied.
I know it is hard, right?:) Hopefully, all this fiddling with compling dlib/pdlib will be gone and replaced with simple sudo apt-get install
. And it will be installed from app store in future... It will still not be as easy as some other apps (as you need to use command line to set up cron to run face recognition...)
I noticed that the photo was not square on the face. Maybe in advanced versions you might be thinking about it :)
Also I did a test,
First step: The application found and marked the faces in the photos and showed on admin panel.
Second Step: After, I've uploaded new photos and fthe application found faces in the new photos but it did not show on the admin panel & details of the photo.
Could this be a bug?
Console:
root@Datarals:/var/www/nextcloud# sudo -u www-data php occ face:background_job -u emresaracoglu -t 4500
1/9 - Executing task CheckRequirementsTask (Check all requirements)
2/9 - Executing task CheckCronTask (Check that service is started from either cron or from command)
3/9 - Executing task LockTask (Acquire lock so that only one background task can run)
4/9 - Executing task StaleImagesRemovalTask (Crawl for stale images (either missing in filesystem or under .nomedia) and remove them from DB)
5/9 - Executing task CreateClustersTask (Create new persons or update existing persons)
Clusters already exist, estimated there is no need to recreate them
6/9 - Executing task AddMissingImagesTask (Crawl for missing images for each user and insert them in DB)
7/9 - Executing task EnumerateImagesMissingFacesTask (Find all images which don't have faces generated for them)
8/9 - Executing task ImageProcessingTask (Process all images to extract faces)
NOTE: Starting face recognition. If you experience random crashes after this point, please look FAQ at https://github.com/matiasdelellis/facerecognition/wiki/FAQ
9/9 - Executing task UnlockTask (Release obtained lock)
Hi @emresaracoglu I'm glad that I'm testing it, and you give us as much feedback as possible.. :smile:
As @stalker314314 commented, It is designed for continuous execution (every 15 minutes?). It can be confusing when you execute it by hand, but in practice it works well.
In short, within the steps, first the grouping is done (Step 5), and then search for new faces (Steps 6, 7 and 8). Therefore, this requires that when you add new photos, in an execution they are analyzed (Find faces, which is the most difficult task), and in the second execution they are grouped.
@matiasdelellis Application doing a good job with CRON. Scanned all faces and clustered!👍 @stalker314314 Thank you for your help for yesterday
Now I have to think about the big question mark for me. How will the current cloud (CentOS) be moved to the new cloud (Ubuntu)?
Not many members are registered (9 Members). The capacity used is 210 GB~ but the files available are small size files (Docs, PDF, etc.) because of the boredom. I can do RSYNC with local IPs, but can you give an idea about the database - how can MySQL database be converted to Postgres DB?
Update //
I hope my plan works properly.
@emresaracoglu I am not expert on Nextcloud (not sure about @matiasdelellis :)) I cannot give you any advise around this, this is huge operation. One hint though: after you move DB and all files, don't forget to run occ files:scan
to rescan all files (and repopulate DB, if needed)
@matiasdelellis This is now 3rd or 4th time that people are complaining about not having clusters after script is done. I had it enough now:) I think it would vastly improve first-time experience if we improve that part. I was thinking to add same task (CreateClustersTask
), but at the end (between ImageProcessingTask
and UnlockTask
). It would not be run always - only if previous task (5/9) didn't find anything. If 5/9 task already did some clusters, this one could be skipped. What do you think, @matiasdelellis?
Hi @emresaracoglu , The migration of servers is quite documented, but the complicated thing is if you change the database.. I'm not sure how to solve that.
You can try to export it in MySQL and inport it with progress but I'm not sure how it will turn out .. If you have 9 users, I understand that you must necessarily migrate the database, so that they keep the shared files, etc. You can not start from scratch .. :disappointed:
@matiasdelellis This is now 3rd or 4th time that people are complaining about not having clusters after script is done. I had it enough now:) I think it would vastly improve first-time experience if we improve that part. I was thinking to add same task (CreateClustersTask), but at the end (between ImageProcessingTask and UnlockTask). It would not be run always - only if previous task (5/9) didn't find anything. If 5/9 task already did some clusters, this one could be skipped. What do you think, @matiasdelellis?
@stalker314314 I guess I was one of the first to discuss this .. :sweat_smile:
This order of operation is justified when executed as a scheduled task. We could consider that it is a scheduled task, when executed limited by timeout? If it is executed without timeout, it means that the admin tries to analyze all the photos and the time does not matter. In this case we can not reverse the tasks to analyze first and group at the end?
@stalker314314 I will use this command if the database fail to restore. @matiasdelellis Thank you for link, very useful.
Therefore, I think we should be able to manually merge the faces (app ask to user: "Are these the same person? they will be merged togetger." if when the same name for face is written by user). I believe this feature will be great if you are the think for the future versions.
I realized that the cluster is not working stably.
The stability of the groups is assured with a greater amount of images... But we try to improve it ..
Also, the application recognizes the same person's face as a different face. This is of course natural.
Unhappily, we must accept some margin of error.. :disappointed:
Therefore, I think we should be able to manually merge the faces (app ask to user: "Are these the same person? they will be merged togetger." if when the same name for face is written by user). I believe this feature will be great if you are the think for the future versions.
This is still in discussion... but for this version we will try to trust that automatic clustering.. and with the feedback of more users we will improve..
I think we should be able to manually merge the faces
Yes, we mentioned that in functional spec, in user operation here, as joining clusters. Not in this version (Google Photos do not have that too:D)
@stalker314314 I guess I was one of the first to discuss this .. sweat_smile
Yes, you are! And you were right. With this amount of evidence + your input, and I changed my mind:) I don't want to be stubborn old man:)
Hi, I understand that this report was resolved, and I close it. Please, if necessary, open it again. :smile:
I have a CentOS 7 operating system installed on my current server, and it works very well but I've created a second virtual machine for your application and installed Ubuntu 18.10 - Nextcloud software.
Beacause I getting this error: CentOS I got this error [view] if not I'm quite satisfied with CentOS 👍
I wrote the command to create a face set on the Ubuntu:
occ face:background_job -u [username]
Click for Console
``` root@server:/var/www/html# sudo -u www-data php occ face:background_job -u [username] An unhandled exception has been thrown: Doctrine\DBAL\DBALException: Failed to connect to the database: An exception occured in driver: could not find driver in /var/www/html/lib/private/DB/Connection.php:64 Stack trace: #0 /var/www/html/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(429): OC\DB\Connection->connect() #1 /var/www/html/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(389): Doctrine\DBAL\Connection->getDatabasePlatformVersion() #2 /var/www/html/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(328): Doctrine\DBAL\Connection->detectDatabasePlatform() #3 /var/www/html/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(623): Doctrine\DBAL\Connection->getDatabasePlatform() #4 /var/www/html/lib/private/DB/Connection.php(151): Doctrine\DBAL\Connection->setTransactionIsolation(2) #5 /var/www/html/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/DriverManager.php(172): OC\DB\Connection->__construct(Array, Object(Doctrine\DBAL\Driver\PDOMySql\Driver), Object(Doctrine\DBAL\Configuration), Object(Doctrine\Common\EventManager)) #6 /var/www/html/lib/private/DB/ConnectionFactory.php(158): Doctrine\DBAL\DriverManager::getConnection(Array, Object(Doctrine\DBAL\Configuration), Object(Doctrine\Common\EventManager)) #7 /var/www/html/lib/private/Server.php(641): OC\DB\ConnectionFactory->getConnection('mysql', Array) #8 /var/www/html/3rdparty/pimple/pimple/src/Pimple/Container.php(118): OC\Server->OC\{closure}(Object(OC\Server)) #9 /var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php(117): Pimple\Container->offsetGet('OCP\\IDBConnecti...') #10 /var/www/html/lib/private/ServerContainer.php(132): OC\AppFramework\Utility\SimpleContainer->query('OCP\\IDBConnecti...') #11 /var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php(165): OC\ServerContainer->query('OCP\\IDBConnecti...') #12 /var/www/html/3rdparty/pimple/pimple/src/Pimple/Container.php(114): OC\AppFramework\Utility\SimpleContainer->OC\AppFramework\Utility\{closure}(Object(OC\Server)) #13 /var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php(117): Pimple\Container->offsetGet('DatabaseConnect...') #14 /var/www/html/lib/private/ServerContainer.php(132): OC\AppFramework\Utility\SimpleContainer->query('DatabaseConnect...') #15 /var/www/html/lib/private/Server.php(1552): OC\ServerContainer->query('DatabaseConnect...') #16 /var/www/html/lib/private/Server.php(360): OC\Server->getDatabaseConnection() #17 /var/www/html/3rdparty/pimple/pimple/src/Pimple/Container.php(118): OC\Server->OC\{closure}(Object(OC\Server)) #18 /var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php(117): Pimple\Container->offsetGet('OC\\Authenticati...') #19 /var/www/html/lib/private/ServerContainer.php(132): OC\AppFramework\Utility\SimpleContainer->query('OC\\Authenticati...') #20 /var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php(68): OC\ServerContainer->query('OC\\Authenticati...') #21 /var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php(98): OC\AppFramework\Utility\SimpleContainer->buildClass(Object(ReflectionClass)) #22 /var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php(119): OC\AppFramework\Utility\SimpleContainer->resolve('OC\\Authenticati...') #23 /var/www/html/lib/private/ServerContainer.php(132): OC\AppFramework\Utility\SimpleContainer->query('OC\\Authenticati...') #24 /var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php(68): OC\ServerContainer->query('OC\\Authenticati...') #25 /var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php(98): OC\AppFramework\Utility\SimpleContainer->buildClass(Object(ReflectionClass)) #26 /var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php(119): OC\AppFramework\Utility\SimpleContainer->resolve('OC\\Authenticati...') #27 /var/www/html/lib/private/ServerContainer.php(132): OC\AppFramework\Utility\SimpleContainer->query('OC\\Authenticati...') #28 /var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php(165): OC\ServerContainer->query('OC\\Authenticati...') #29 /var/www/html/3rdparty/pimple/pimple/src/Pimple/Container.php(114): OC\AppFramework\Utility\SimpleContainer->OC\AppFramework\Utility\{closure}(Object(OC\Server)) #30 /var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php(117): Pimple\Container->offsetGet('OC\\Authenticati...') #31 /var/www/html/lib/private/ServerContainer.php(132): OC\AppFramework\Utility\SimpleContainer->query('OC\\Authenticati...') #32 /var/www/html/lib/private/Server.php(372): OC\ServerContainer->query('OC\\Authenticati...') #33 /var/www/html/3rdparty/pimple/pimple/src/Pimple/Container.php(118): OC\Server->OC\{closure}(Object(OC\Server)) #34 /var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php(117): Pimple\Container->offsetGet('OC\\User\\Session') #35 /var/www/html/lib/private/ServerContainer.php(132): OC\AppFramework\Utility\SimpleContainer->query('OC\\User\\Session') #36 /var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php(165): OC\ServerContainer->query('OC\\User\\Session') #37 /var/www/html/3rdparty/pimple/pimple/src/Pimple/Container.php(114): OC\AppFramework\Utility\SimpleContainer->OC\AppFramework\Utility\{closure}(Object(OC\Server)) #38 /var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php(117): Pimple\Container->offsetGet('UserSession') #39 /var/www/html/lib/private/ServerContainer.php(132): OC\AppFramework\Utility\SimpleContainer->query('UserSession') #40 /var/www/html/lib/private/Server.php(1428): OC\ServerContainer->query('UserSession') #41 /var/www/html/lib/private/Server.php(694): OC\Server->getUserSession() #42 /var/www/html/3rdparty/pimple/pimple/src/Pimple/Container.php(118): OC\Server->OC\{closure}(Object(OC\Server)) #43 /var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php(117): Pimple\Container->offsetGet('OC\\App\\AppManag...') #44 /var/www/html/lib/private/ServerContainer.php(132): OC\AppFramework\Utility\SimpleContainer->query('OC\\App\\AppManag...') #45 /var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php(165): OC\ServerContainer->query('OC\\App\\AppManag...') #46 /var/www/html/3rdparty/pimple/pimple/src/Pimple/Container.php(114): OC\AppFramework\Utility\SimpleContainer->OC\AppFramework\Utility\{closure}(Object(OC\Server)) #47 /var/www/html/lib/private/AppFramework/Utility/SimpleContainer.php(117): Pimple\Container->offsetGet('AppManager') #48 /var/www/html/lib/private/ServerContainer.php(132): OC\AppFramework\Utility\SimpleContainer->query('AppManager') #49 /var/www/html/lib/private/Server.php(1723): OC\ServerContainer->query('AppManager') #50 /var/www/html/lib/private/legacy/app.php(342): OC\Server->getAppManager() #51 /var/www/html/lib/private/legacy/app.php(113): OC_App::getEnabledApps() #52 /var/www/html/lib/base.php(654): OC_App::loadApps(Array) #53 /var/www/html/lib/base.php(1068): OC::init() #54 /var/www/html/console.php(46): require_once('/var/www/html/l...') #55 /var/www/html/occ(11): require_once('/var/www/html/c...') #56 {main}PHP Fatal error: Uncaught Doctrine\DBAL\DBALException: Failed to connect to the database: An exception occured in driver: could not find driver in /var/www/html/lib/private/DB/Connection.php:64 Stack trace: #0 /var/www/html/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(429): OC\DB\Connection->connect() #1 /var/www/html/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(389): Doctrine\DBAL\Connection->getDatabasePlatformVersion() #2 /var/www/html/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(328): Doctrine\DBAL\Connection->detectDatabasePlatform() #3 /var/www/html/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(623): Doctrine\DBAL\Connection->getDatabasePlatform() #4 /var/www/html/lib/private/DB/Connection.php(151): Doctrine\DBAL\Connection->setTransactionIsolation(2) #5 /var/www/html/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/DriverManager.php(172): OC\DB\Connection->__construct(Array, Object(Doctrine\DBAL\Driver\PDOMySql\Driver), Object(Doctrine\DBAL\Configuration), Object(Doctrine\Common\Event in /var/www/html/lib/private/DB/Connection.php on line 64 ```