Fast Automation: Fast Scraping of Incoming Messages + Fast Deliver + Full-Automated Delivery

[ExpandedVenture]

Problem

Some incoming messages are being scraped too late.

Solution

Dispatch an inbox scraping job when the user receives an email notification of an incoming message.

• verify all users are configured to receive individual emails of incoming messages (not summaries),
• create signals table,
• use the signals table to decide to dispatch a scrape.inbox job.

[ExpandedVenture]

• fast incoming messages scraping

[ExpandedVenture]

• IDEA: assign a worker to an account when a new signal arrives? --> no

[ExpandedVenture]

Merged with #87.

[ExpandedVenture]

ANALYSIS & DESIGN

1. signal of incoming message + show signal in the responses screen
2. dispatch.recover.messages
3. send messages and send invites when scrape.inbox
4. send messages when running ininvite.result also
5. dispatch for messaging in the moment when a manual message is created
6. automated process to check that signals for all the accounts are workking + query for daily check
7. query for daily check both, average and max time, from the moment of an incoming message to the moment of that message have been scraped
8. query for daily check both, average and max time, from the moment of a message has been created to the moment of that message have been delivered

[leandrosardi]

1. signal of incoming message

use copernico;
go

select recover_end_time, stat_last_signal_incoming_message, stat_last_signal_accepted_connection_request
from lnuser u
where u.delete_time is null;
go

-- deprecated
drop table signal;
go

create table signal (
    id uniqueidentifier not null primary key,
    create_time datetime not null,
    [type] int not null, -- 0:linkedin accepted connection requests, 1: linkedin incoming message
    [to] varchar(500) not null, -- receipt email
    id_lnuser uniqueidentifier null references lnuser([id]), -- it is null if the social account is not registered into the platform
    id_result uniqueidentifier null references result([id]), -- it is null if the LinkedIn user is not belonging a prospecting campaign
    [subject] varchar(8000) not null,
    html text not null,
);
go

[leandrosardi]

2. dispatch.recover.messages

• fix dispatcher

[leandrosardi]

3. bundle all the tasks into 1 single process

1. refactorization of all the files inside the folder /p/;
2. moving the pampa processes from the bots gem to the tempora project;
3. bundle all the prospecting operations (repair, recover, invite, message) into 1 single pampa process;
4. add access points to ask if it is necessary to scrape inbox;
5. new cli folder in the tempora project instead the bots project.
6. all dispatchers regarding prospecting should handle new reservaton fields:

alter table lnuser add prospecting_reservation_id uniqueidentifier null;
alter table lnuser add prospecting_reservation_time datetime null;
alter table lnuser add prospecting_start_time datetime null;
alter table lnuser add prospecting_end_time datetime null;
go

create index IX_lnuser__prospecting_reservation_id on lnuser(prospecting_reservation_id);
go

drop index IX_lnuser__ininvite_reservation_id on lnuser;
go

alter table lnuser drop column inmessage_reservation_id;
alter table lnuser drop column inmessage_reservation_time;
go

alter table lnuser drop column ininvite_reservation_id;
alter table lnuser drop column ininvite_reservation_time;
go

alter table lnuser drop column repair_reservation_id;
alter table lnuser drop column repair_reservation_time;
go

alter table lnuser drop column recover_reservation_id;
alter table lnuser drop column recover_reservation_time;
go

• Develop this: # actualizo el id_object para registrar en la tabla boterrorlog, en caso de que ocurra un error.

• Develop this: window = 300 # 5 minutes

• Develop this: api1.3/pampa/prospecting/

• Develop this: get_lnuser

• self.notifyLnUserStatus(lnuser['id'], BlackStack::BaseLnUser::LNUSER_STATUS_NEW_USER_INTERFACE)

el lnuser debe hacer un ping a la division

self.logger.logs("Ping to Division... ") self.ping() self.logger.done

notify LnUser is working

self.logger.logs("Updating LnUser status... ") self.notifyLnUserStatus(lnuser['id'], BlackStack::BaseLnUser::LNUSER_STATUS_NEW_USER_INTERFACE) # actualiza last_signal_time, para que el dispatcher sepa que este lnuser esta siendo utilizado por un worker self.logger.done

WRITE PROCESS

WRITE APP.RB

WRITE DISPATCHERS

MOVE & REWRITE ACCESS POINTS

repair:

notify_repair_start notify_repair_finish getNewEmailAddress --> verify it returns nil when there is not new email, but don't raise exteption

recover:

notify_scrape_start notify_scrape_finish TODO: need_to_scrape notify_lnchat.json

inchat.result:

getChats getChat notifyInChatStart notifyInChatFinish

notifyInChatResult

-- getProfilesForInCrawl notifyInCrawlProfileStart notifyInCrawlProfileFinish sendProfileInfo

ininvite.result:

convert_sales_navigator_profile_to_linkedin_profile get_results get_result notify_ininvite_result notify_ininvite_start notify_ininvite_finish notify_ininvite_info

[ExpandedVenture]

4. Large Refactorization

thr --> threads main --> deleted drl --> prospecting drxdr --> prospecting ssm3/ --> prospecting/pipeline/ blast/ --> prospecting/campaign/ crowd --> crowdtrust crm --> (deleted) edb --> listbuilder lgb --> (deleted) knowledge --> (deleted) mac --> farming em5 --> emailservice ef --> emailaddresses main --> (deleted) member --> settings shm --> hostsmonitoring

api1.4 --> (deleted) api1.5 --> (deleted) api1.6 --> (deleted)

api1.4/scl/ --> api1.3/threads/ api1.4/shm --> api1.3/hostsmonitoring

api1.3/sonar --> (deleted) api1.3/drx --> api1.3/prospecting api1.3/edb --> api1.3/listbuilder api1.3/em5 --> api1.3/emailservice api1.3/ef --> api1.3/emailaddresses api1.3/remote --> api1.3/crowdtrust api1.3/pampa --> api1.3/bots api1.3/pampa/create.lnuser --> api1.3/bots/farming api1.3/pampa/incrawl.lnseaerchvariation --> api1.3/bots/listbuilder api1.3/pampa/remote --> api1.3/bots/crowdtrust

[ExpandedVenture]

Refactorization of the processes names

update kepler..worker set assigned_process='./p/prospecting.rb' where assigned_process='./p/botting.rb';
update kepler..worker set assigned_process='./p/_dispatcher.prospecting.rb' where assigned_process='./p/_dispatcher.rb';
update kepler..worker set assigned_process='./p/_dispatcher.crowdtrust.rb' where assigned_process='./p/_dispatcher_crowdtrust.rb';

update euler..worker set assigned_process='./p/prospecting.rb' where assigned_process='./p/botting.rb';
update euler..worker set assigned_process='./p/_dispatcher.prospecting.rb' where assigned_process='./p/_dispatcher.rb';
update euler..worker set assigned_process='./p/_dispatcher.crowdtrust.rb' where assigned_process='./p/_dispatcher_crowdtrust.rb';

[ExpandedVenture]

Refactorization of the processes names

update kepler..worker set assigned_process='./p/sub_dispatcher.prospecting.rb' where assigned_process='./p/dispatcher.rb';
update kepler..worker set assigned_process='./p/sub_dispatcher.prospecting.rb' where assigned_process='./p/dispatcher.rb';

update kepler..worker set assigned_process='./p/bots.prospecting.rb' where assigned_process='./p/prospecting.rb';
update euler..worker set assigned_process='./p/bots.prospecting.rb' where assigned_process='./p/prospecting.rb';

-- fix i2p processing
update kepler..worker set process='./p/_dispatcher.i2p.rb', assigned_process='./p/_dispatcher.i2p.rb' where [name]='KEPLER.00-50-56-01-16-16-14.dispatcher_kepler'
update euler..worker set process='./p/_dispatcher.i2p.rb', assigned_process='./p/_dispatcher.i2p.rb' where [name]='KEPLER.00-50-56-01-16-16-14.dispatcher_kepler'

-- fix buffer.listbuilder
update kepler..worker set process='./p/buffer.listbuilder.rb', assigned_process='./p/buffer.listbuilder.rb' where assigned_process='./p/copy.scrape.block.rb';
update euler..worker set process='./p/buffer.listbuilder.rb', assigned_process='./p/buffer.listbuilder.rb' where assigned_process='./p/copy.scrape.block.rb';