search

ageitgey / face_recognition

The world's simplest facial recognition api for Python and the command line
MIT License
52.84k stars 13.43k forks source link

How to use KNN with Apache Solr? #828

Open Dentrax opened 5 years ago

Dentrax commented 5 years ago

Previously, we were using this system when we were recogniting multiple faces for per person. But now, we have switched the database system following the #238 (BTW, Thanks @khaledabbad for recommended us to use Apache Solr) issue and the results and speeds are amazing!

Now, the system works flawlessly for single-trained face if this structure is being used:

     Structure:
        <train_dir>/
        ├── <person1>/
        │   ├── <single>.jpeg
        ├── <person2>/
        │   ├── <single>.jpeg
        ├── <person3>/
        │   ├── <single>.jpeg
        └── ...

If we have datas like this structure:

     Structure:
        <train_dir>/
        ├── <person1>/
        │   ├── <somename1>.jpeg
        │   ├── <somename2>.jpeg
        │   ├── ...
        ├── <person2>/
        │   ├── <somename1>.jpeg
        │   └── <somename2>.jpeg
        └── ...

... the system does not learn a single person with more than one face. Because we have not implemented KNNalgorithm yet.

The question is: How can we use the KNN algorithm with this structure (using with 128-column database design) ?

tomriddle54 commented 5 years ago

@Dentrax can you share the code your extracting encodings and saving it to db + Euclidean distance code to get results. maybe I can try what you want and share you the feedback

Dentrax commented 5 years ago

@syedmustafa54 We just used pysolr, I think you should take a look. ;)

Any ideas? :) @ageitgey @itamar8910 @AlainPilon @psyapathy

anigogo commented 5 years ago

@Dentrax can you share the source code if possible

Thanks

tomriddle54 commented 5 years ago

@Dentrax can you tell how are you query I have added 128 points in solr for two images id1,id2,id3,id4,id5,id6,id7,id8,id9,id10,id11,id12,id13,id14,id15,id16,id17,id18,id19,id20,id21,id22,id23,id24,id25,id26,id27,id28,id29,id30,id31,id32,id33,id34,id35,id36,id37,id38,id39,id40,id41,id42,id43,id44,id45,id46,id47,id48,id49,id50,id51,id52,id53,id54,id55,id56,id57,id58,id59,id60,id61,id62,id63,id64,id65,id66,id67,id68,id69,id70,id71,id72,id73,id74,id75,id76,id77,id78,id79,id80,id81,id82,id83,id84,id85,id86,id87,id88,id89,id90,id91,id92,id93,id94,id95,id96,id97,id98,id99,id100,id101,id102,id103,id104,id105,id106,id107,id108,id109,id110,id111,id112,id113,id114,id115,id116,id117,id118,id119,id120,id121,id122,id123,id124,id125,id126,id127,id128

asamedeiros commented 4 years ago

Previously, we were using this system when we were recogniting multiple faces for per person. But now, we have switched the database system following the #238 (BTW, Thanks @khaledabbad for recommended us to use Apache Solr) issue and the results and speeds are amazing!

Now, the system works flawlessly for single-trained face if this structure is being used:

     Structure:
        <train_dir>/
        ├── <person1>/
        │   ├── <single>.jpeg
        ├── <person2>/
        │   ├── <single>.jpeg
        ├── <person3>/
        │   ├── <single>.jpeg
        └── ...

If we have datas like this structure:

     Structure:
        <train_dir>/
        ├── <person1>/
        │   ├── <somename1>.jpeg
        │   ├── <somename2>.jpeg
        │   ├── ...
        ├── <person2>/
        │   ├── <somename1>.jpeg
        │   └── <somename2>.jpeg
        └── ...

... the system does not learn a single person with more than one face. Because we have not implemented KNNalgorithm yet.

The question is: How can we use the KNN algorithm with this structure (using with 128-column database design) ?

@Dentrax can you share the source code if possible?

Thanks

Dentrax commented 4 years ago

You can do it yourself with a short research. Please learn to do research. Useful links:

1) Theory 2) API Request Example 3) PySolr

Thanks

asamedeiros commented 4 years ago

Could you at least give me this full string? Only this part I can't.

http://localhost:8983/solr/mycore/select?q=:&fl=dist(2,v_0,v_1,v_3,.. <http://localhost:8983/solr/mycore/select?q=:&fl=dist(2,v_0,v_1,v_3,..>.,v_127,-0.0621345,0.048437204,0.0839613,...)

Thanks

Em sex, 20 de set de 2019 às 08:08, Furkan Türkal notifications@github.com escreveu:

Sorry, I can't. You can do it yourself with a short research. Please learn to do research. Useful links:

  1. Theory https://github.com/ageitgey/face_recognition/issues/238#issuecomment-345847465
  2. API Request Example https://github.com/ageitgey/face_recognition/issues/238#issuecomment-346010835
  3. PySolr https://github.com/django-haystack/pysolr

Thanks

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/ageitgey/face_recognition/issues/828?email_source=notifications&email_token=ABIMVQQ7552H6QNVYPA47DTQKSVKJA5CNFSM4HPKMBN2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7GLPDY#issuecomment-533510031, or mute the thread https://github.com/notifications/unsubscribe-auth/ABIMVQS3O6VPDJ4BEJSV3XTQKSVKJANCNFSM4HPKMBNQ .

Dentrax commented 4 years ago

You must generate your own string with for-loop using your custom distances. Like:

self.SOLR_QUERY_DIST = 'dist(2,'
 for i in range(0, 128):
  self.SOLR_QUERY_DIST += 'v_{},'.format(str(i))

To generate custom values:

query = self.SOLR_QUERY_DIST
 for i, val in enumerate(faces_encodings):
  query += '{},'.format(str(val))
 query += ')'
asamedeiros commented 4 years ago

Thank you very much!!!

Em dom, 22 de set de 2019 às 13:25, Furkan Türkal notifications@github.com escreveu:

You must generate your own string with for-loop using your custom distances. Like:

self.SOLR_QUERY_DIST = 'dist(2,' for i in range(0, 128): self.SOLR_QUERYDIST += 'v{},'.format(str(i))

To generate custom values:

query = self.SOLR_QUERY_DIST for i, val in enumerate(faces_encodings): query += '{},'.format(str(val)) query += ')'

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/ageitgey/face_recognition/issues/828?email_source=notifications&email_token=ABIMVQQ47MREDR4A45SAP3TQK6MBJA5CNFSM4HPKMBN2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7JJWXY#issuecomment-533896031, or mute the thread https://github.com/notifications/unsubscribe-auth/ABIMVQR5U66JQ6SA4FFWBLDQK6MBJANCNFSM4HPKMBNQ .

alessiosavi commented 4 years ago

Euclidean distance or KNN are not very useful when dealing with image recognition.

You can use a MLP neural network in order to train/predict a lot of images.

I've used this approach in PyRecognizer. The time to predict the face is instant (few milliseconds) with a dataset of 311 people (celebrities), and a total of 5425 photos.

PonraJS-21 commented 3 years ago

Previously, we were using this system when we were recogniting multiple faces for per person. But now, we have switched the database system following the #238 (BTW, Thanks @khaledabbad for recommended us to use Apache Solr) issue and the results and speeds are amazing!

Now, the system works flawlessly for single-trained face if this structure is being used:

     Structure:
        <train_dir>/
        ├── <person1>/
        │   ├── <single>.jpeg
        ├── <person2>/
        │   ├── <single>.jpeg
        ├── <person3>/
        │   ├── <single>.jpeg
        └── ...

If we have datas like this structure:

     Structure:
        <train_dir>/
        ├── <person1>/
        │   ├── <somename1>.jpeg
        │   ├── <somename2>.jpeg
        │   ├── ...
        ├── <person2>/
        │   ├── <somename1>.jpeg
        │   └── <somename2>.jpeg
        └── ...

... the system does not learn a single person with more than one face. Because we have not implemented KNNalgorithm yet.

The question is: How can we use the KNN algorithm with this structure (using with 128-column database design) ?

Could you please tell me what field type you used in Apache solr?