search

fandangOrg / fandango

FAke News discovery and propagation from big Data ANalysis and artificial intelliGence Operations
1 stars 1 forks source link

Asynchronous data ingestion #69

Closed pstalidis closed 3 years ago

pstalidis commented 4 years ago

Right now, data ingestion is a synchronous process. This means that all analysis must finish before an article can be ingested in the database. This means that ingestion can fail if any analyser fails (single pint of failure) and the speed of ingestion is limited to the slowest of the analysers. Since each part of the analysis is stored separately (fdg-entity, fdg-media, fdg-author, etc), each of the analysers should return a promise (the id in the appropriate index) of where the analysis will be stored and perform the actual analysis in a separate thread. There are 2 problems (that I can think of) that need to be solved for this:

  1. the text modality has to be separated from the article entity (to be consistent with other modalities)
  2. we need a notification system for the process that produces an article score that all modalities have been analysed
jefersonzanim commented 4 years ago

Hi @pstalidis ! Is there anything I can help directly with this?

danielevannella commented 4 years ago

Hello @pstalidis why did you assign me this?

danielevannella commented 4 years ago

Sorry I didn't see all.

pstalidis commented 4 years ago

@danielevannella I assigned everyone involved in the data ingestion process @jefersonzanim we should discuss this issue in the next technical call

AlbertoGhedin commented 4 years ago

Agree, let discuss in next call

dmgutierrez commented 4 years ago

Hi Guys,

The next technical call is planned to be held this Friday October 4th or is the one that we have every two weeks on Tuesdays?

On Wed, Oct 2, 2019 at 1:55 PM AlbertoGhedin notifications@github.com wrote:

Agree, let discuss in next call

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/fandangOrg/fandango/issues/69?email_source=notifications&email_token=AGLEGKBJIOUNHVGPIOEGLXLQMSD2ZA5CNFSM4I352FD2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAEPMJA#issuecomment-537458212, or mute the thread https://github.com/notifications/unsubscribe-auth/AGLEGKG6CFRINGVOIQRJWFDQMSD2ZANCNFSM4I352FDQ .

jefersonzanim commented 4 years ago

We have a call on Friday. All topics pertinent to the execution of the Pilots can be addressed then.

Jeferson Zanim Head of Client Services, Siren A Block C, 77 Sir John Rogerson's Quay, Dublin, D02 T804, Ireland P +353 (0)1 553 0200 <+353+(0)1+553+0200> M +353 85 107 7810 <+353+85+107+7810> E jeferson.zanim@siren.io jeferson.zanim@siren.io W www.siren.io http://www.siren.io?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature https://twitter.com/sirensearch?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature https://www.youtube.com/channel/UCKGsC-vD28r7hW6T9QspKPA?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature https://vimeo.com/sirenio?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature https://www.facebook.com/sirensearch?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature https://www.linkedin.com/company/11117365?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_content=&utm_campaign=signature

On Wed, 2 Oct 2019 at 15:09, David Martín Gutiérrez < notifications@github.com> wrote:

Hi Guys,

The next technical call is planned to be held this Friday October 4th or is the one that we have every two weeks on Tuesdays?

On Wed, Oct 2, 2019 at 1:55 PM AlbertoGhedin notifications@github.com wrote:

Agree, let discuss in next call

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub < https://github.com/fandangOrg/fandango/issues/69?email_source=notifications&email_token=AGLEGKBJIOUNHVGPIOEGLXLQMSD2ZA5CNFSM4I352FD2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAEPMJA#issuecomment-537458212 , or mute the thread < https://github.com/notifications/unsubscribe-auth/AGLEGKG6CFRINGVOIQRJWFDQMSD2ZANCNFSM4I352FDQ

.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/fandangOrg/fandango/issues/69?email_source=notifications&email_token=AJA56S3NRDH6SZHEGKBSFTDQMSTRTA5CNFSM4I352FD2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAE4BSY#issuecomment-537510091, or mute the thread https://github.com/notifications/unsubscribe-auth/AJA56S3LH2ILOS6ZW4MWW7LQMSTRTANCNFSM4I352FDQ .

mmagaldi-eng commented 4 years ago

I'm sorry guys, but I cannot attend the conf call tomorrow. We should discuss about it next week (I've just sent an email on this). As I've written, to reduce reworking we should consider this together with fusion score (T4.5).

mmagaldi-eng commented 4 years ago

@danielevannella I assigned everyone involved in the data ingestion process @jefersonzanim we should discuss this issue in the next technical call

@pstalidis don't forget that (despite myself) I'm the person in charge of the "offline" process... ;)

jefersonzanim commented 4 years ago

@mmagaldi-eng there's no technical call currently scheduled.

pstalidis commented 4 years ago

@mmagaldi-eng sorry, I thought I had assigned you too @jefersonzanim FANDANGO - Technical Call Tue, Oct 8, 2019 10:30 PM - 12:30 AM CEST Please join my meeting from your computer, tablet or smartphone. https://global.gotomeeting.com/join/990869773

danielevannella commented 4 years ago

I'll be at a conference.

Il giorno ven 4 ott 2019 alle ore 15:36 Panagiotis Stalidis < notifications@github.com> ha scritto:

@mmagaldi-eng https://github.com/mmagaldi-eng sorry, I thought I had assigned you too @jefersonzanim https://github.com/jefersonzanim FANDANGO - Technical Call Tue, Oct 8, 2019 10:30 PM - 12:30 AM CEST Please join my meeting from your computer, tablet or smartphone. https://global.gotomeeting.com/join/990869773

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/fandangOrg/fandango/issues/69?email_source=notifications&email_token=AJCVVNITBZPCZ7TGYHVYEJDQM5BFJA5CNFSM4I352FD2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEALVHII#issuecomment-538399649, or mute the thread https://github.com/notifications/unsubscribe-auth/AJCVVNPEQ3LXEJ3MLT6OYGDQM5BFJANCNFSM4I352FDQ .

pstalidis commented 4 years ago

ingestion has been postponed for after the pilot execution

pstalidis commented 3 years ago

This should have been closed a long time ago