glitch: dropbox-client error `path/not_found` --> Migrate from DropBox to AWS S3

leandrosardi commented 1 month ago

ruby profile.rb api_key=07e17fd3-6af1-4bf9-b2bf-faed2c81f51f api_url=https://s01.massprospecting.com:443 id=5fa0ad4b-5797-4470-baf4-fa9ebc09f7f1 ignore-hostname=yes force-profile-run=yes headless=no run-once=yes inboxcheck=no connectioncheck=no scraping=yes id_job=5c2e51f7-fe05-49fe-808a-af958851f271 enrichment=no outreach=no

Error: path/not_found/..
/usr/local/rvm/gems/ruby-3.1.2/gems/my-dropbox-api-1.0.2/lib/my-dropbox-api.rb:227:in `get_file_url'
/usr/local/rvm/gems/ruby-3.1.2/gems/mass-client-1.0.20/lib/first-line/profile_rpa.rb:278:in `download_image_0'
/usr/local/rvm/gems/ruby-3.1.2/gems/mass-client-1.0.20/lib/first-line/profile_rpa.rb:289:in `download_image'
/home/blackstack/code1/sdk/lib/source_types/LinkedIn_PublicFeed.rb:251:in `block (2 levels) in do'
/home/blackstack/code1/sdk/lib/source_types/LinkedIn_PublicFeed.rb:251:in `map'
/home/blackstack/code1/sdk/lib/source_types/LinkedIn_PublicFeed.rb:251:in `block in do'
/home/blackstack/code1/sdk/lib/source_types/LinkedIn_PublicFeed.rb:165:in `each'
/home/blackstack/code1/sdk/lib/source_types/LinkedIn_PublicFeed.rb:165:in `do'
/home/blackstack/code1/sdk/p/profile.rb:772:in `block in 
'
/home/blackstack/code1/sdk/p/profile.rb:764:in `each'
/home/blackstack/code1/sdk/p/profile.rb:764:in `
'

leandrosardi commented 1 month ago

After Latest Patches

Dropbox file upload failed. Dropbox response: {"error_summary": "too_many_write_operations/...", "error": {"reason": {".tag": "too_many_write_operations"}, "retry_after": 1}}
/usr/local/rvm/gems/ruby-3.1.2/gems/mass-client-1.0.23/lib/first-line/profile_rpa.rb:333:in `download_image_0'
/usr/local/rvm/gems/ruby-3.1.2/gems/mass-client-1.0.23/lib/first-line/profile_rpa.rb:353:in `download_image'
/home/blackstack/code1/sdk/lib/source_types/LinkedIn_PublicFeed.rb:261:in `block (2 levels) in do'
/home/blackstack/code1/sdk/lib/source_types/LinkedIn_PublicFeed.rb:261:in `map'
/home/blackstack/code1/sdk/lib/source_types/LinkedIn_PublicFeed.rb:261:in `block in do'
/home/blackstack/code1/sdk/lib/source_types/LinkedIn_PublicFeed.rb:165:in `each'
/home/blackstack/code1/sdk/lib/source_types/LinkedIn_PublicFeed.rb:165:in `do'
/home/blackstack/code1/sdk/p/profile.rb:772:in `block in 
'
/home/blackstack/code1/sdk/p/profile.rb:764:in `each'
/home/blackstack/code1/sdk/p/profile.rb:764:in `
'

leandrosardi commented 1 month ago

Another Problem

The too_many_write_operations is happening at the server side too.

Error calling upsert endpoint: Error: path/not_found/
/usr/local/rvm/gems/ruby-3.1.2/gems/my-dropbox-api-1.0.2/lib/my-dropbox-api.rb:227:in `get_file_url'
/home/blackstack/code1/slave/lib/skeletons.rb:247:in `store'
/home/blackstack/code1/slave/extensions/mass.subaccount/models/skeleton/lead.rb:251:in `update'
/home/blackstack/code1/slave/lib/skeletons.rb:61:in `insert'
/home/blackstack/code1/slave/lib/skeletons.rb:76:in `upsert'
/home/blackstack/code1/slave/extensions/mass.subaccount/models/skeleton/event.rb:112:in `find'
/home/blackstack/code1/slave/lib/skeletons.rb:73:in `upsert'
/home/blackstack/code1/slave/views/api1.0/upsert.erb:14:in `__tilt_4449380'
/usr/local/rvm/gems/ruby-3.1.2/gems/tilt-2.4.0/lib/tilt/template.rb:207:in `bind_call'
/usr/local/rvm/gems/ruby-3.1.2/gems/tilt-2.4.0/lib/tilt/template.rb:207:in `evaluate'
/usr/local/rvm/gems/ruby-3.1.2/gems/tilt-2.4.0/lib/tilt/template.rb:102:in `render'
/usr/local/rvm/gems/ruby-3.1.2/gems/sinatra-2.2.4/lib/sinatra/base.rb:863:in `render'
/usr/local/rvm/gems/ruby-3.1.2/gems/sinatra-2.2.4/lib/sinatra/base.rb:711:in `erb'
/home/blackstack/code1/slave/app.rb:667:in `block in 
'
/usr/local/rvm/gems/ruby-3.1.2/gems/sinatra-2.2.4/lib/sinatra/base.rb:1697:in `call'
/usr/local/rvm/gems/ruby-3.1.2/gems/sinatra-2.2.4/lib/sinatra/base.rb:1697:in `block in compile!'
/usr/local/rvm/gems/ruby-3.1.2/gems/sinatra-2.2.4/lib/sinatra/base.rb:1030:in `block (3 levels) in route!'
/usr/local/rvm/gems/ruby-3.1.2/gems/sinatra-2.2.4/lib/sinatra/base.rb:1049:in `route_eval'
/usr/local/rvm/gems/ruby-3.1.2/gems/sinatra-2.2.4/lib/sinatra/base.rb:1030:in `block (2 levels) in route!'
/usr/local/rvm/gems/ruby-3.1.2/gems/sinatra-2.2.4/lib/sinatra/base.rb:1078:in `block in process_route'
/usr/local/rvm/gems/ruby-3.1.2/gems/sinatra-2.2.4/lib/sinatra/base.rb:1076:in `catch'
/usr/local/rvm/gems/ruby-3.1.2/gems/sinatra-2.2.4/lib/sinatra/base.rb:1076:in `process_route'
/usr/local/rvm/gems/ruby-3.1.2/gems/sinatra-2.2.4/lib/sinatra/base.rb:1028:in `block in route!'
/usr/local/rvm/gems/ruby-3.1.2/gems/sinatra-2.2.4/lib/sinatra/base.rb:1025:in `each'
/usr/local/rvm/gems/ruby-3.1.2/gems/sinatra-2.2.4/lib/sinatra/base.rb:1025:in `route!'
/usr/local/rvm/gems/ruby-3.1.2/gems/sinatra-2.2.4/lib/sinatra/base.rb:1147:in `block in dispatch!'
/usr/local/rvm/gems/ruby-3.1.2/gems/sinatra-2.2.4/lib/sinatra/base.rb:1119:in `block in invoke'
/usr/local/rvm/gems/ruby-3.1.2/gems/sinatra-2.2.4/lib/sinatra/base.rb:1119:in `catch'
/usr/local/rvm/gems/ruby-3.1.2/gems/sinatra-2.2.4/lib/sinatra/base.rb:1119:in `invoke'
/usr/local/rvm/gems/ruby-3.1.2/gems/sinatra-2.2.4/lib/sinatra/base.rb:1142:in `dispatch!'
/usr/local/rvm/gems/ruby-3.1.2/gems/sinatra-2.2.4/lib/sinatra/base.rb:956:in `block in call!'
/usr/local/rvm/gems/ruby-3.1.2/gems/sinatra-2.2.4/lib/sinatra/base.rb:1119:in `block in invoke'
/usr/local/rvm/gems/ruby-3.1.2/gems/sinatra-2.2.4/lib/sinatra/base.rb:1119:in `catch'
/usr/local/rvm/gems/ruby-3.1.2/gems/sinatra-2.2.4/lib/sinatra/base.rb:1119:in `invoke'
/usr/local/rvm/gems/ruby-3.1.2/gems/sinatra-2.2.4/lib/sinatra/base.rb:956:in `call!'
/usr/local/rvm/gems/ruby-3.1.2/gems/sinatra-2.2.4/lib/sinatra/base.rb:945:in `call'
/usr/local/rvm/gems/ruby-3.1.2/gems/rack-contrib-2.3.0/lib/rack/contrib/try_static.rb:35:in `call'
/usr/local/rvm/gems/ruby-3.1.2/gems/rack-contrib-2.3.0/lib/rack/contrib/try_static.rb:35:in `call'
/usr/local/rvm/gems/ruby-3.1.2/gems/rack-contrib-2.3.0/lib/rack/contrib/try_static.rb:35:in `call'
/usr/local/rvm/gems/ruby-3.1.2/gems/rack-contrib-2.3.0/lib/rack/contrib/try_static.rb:35:in `call'
/usr/local/rvm/gems/ruby-3.1.2/gems/rack-contrib-2.3.0/lib/rack/contrib/try_static.rb:35:in `call'
/usr/local/rvm/gems/ruby-3.1.2/gems/rack-2.2.10/lib/rack/session/abstract/id.rb:266:in `context'
/usr/local/rvm/gems/ruby-3.1.2/gems/rack-2.2.10/lib/rack/session/abstract/id.rb:260:in `call'
/usr/local/rvm/gems/ruby-3.1.2/gems/rack-2.2.10/lib/rack/logger.rb:17:in `call'
/usr/local/rvm/gems/ruby-3.1.2/gems/rack-2.2.10/lib/rack/common_logger.rb:38:in `call'
/usr/local/rvm/gems/ruby-3.1.2/gems/sinatra-2.2.4/lib/sinatra/base.rb:255:in `call'
/usr/local/rvm/gems/ruby-3.1.2/gems/sinatra-2.2.4/lib/sinatra/base.rb:248:in `call'
/usr/local/rvm/gems/ruby-3.1.2/gems/rack-2.2.10/lib/rack/head.rb:12:in `call'
/usr/local/rvm/gems/ruby-3.1.2/gems/rack-2.2.10/lib/rack/method_override.rb:24:in `call'
/usr/local/rvm/gems/ruby-3.1.2/gems/sinatra-2.2.4/lib/sinatra/show_exceptions.rb:22:in `call'
/usr/local/rvm/gems/ruby-3.1.2/gems/sinatra-2.2.4/lib/sinatra/base.rb:218:in `call'
/usr/local/rvm/gems/ruby-3.1.2/gems/sinatra-2.2.4/lib/sinatra/base.rb:2004:in `call'
/usr/local/rvm/gems/ruby-3.1.2/gems/sinatra-2.2.4/lib/sinatra/base.rb:1564:in `block in call'
/usr/local/rvm/gems/ruby-3.1.2/gems/sinatra-2.2.4/lib/sinatra/base.rb:1780:in `synchronize'
/usr/local/rvm/gems/ruby-3.1.2/gems/sinatra-2.2.4/lib/sinatra/base.rb:1564:in `call'
/usr/local/rvm/gems/ruby-3.1.2/gems/puma-5.6.9/lib/puma/configuration.rb:252:in `call'
/usr/local/rvm/gems/ruby-3.1.2/gems/puma-5.6.9/lib/puma/request.rb:77:in `block in handle_request'
/usr/local/rvm/gems/ruby-3.1.2/gems/puma-5.6.9/lib/puma/thread_pool.rb:340:in `with_force_shutdown'
/usr/local/rvm/gems/ruby-3.1.2/gems/puma-5.6.9/lib/puma/request.rb:76:in `handle_request'
/usr/local/rvm/gems/ruby-3.1.2/gems/puma-5.6.9/lib/puma/server.rb:443:in `process_client'
/usr/local/rvm/gems/ruby-3.1.2/gems/puma-5.6.9/lib/puma/thread_pool.rb:147:in `block in spawn_thread'
/usr/local/rvm/gems/ruby-3.1.2/gems/blackstack-core-1.2.28/lib/functions.rb:247:in `upsert'
/usr/local/rvm/gems/ruby-3.1.2/gems/blackstack-core-1.2.28/lib/functions.rb:254:in `upsert'
/usr/local/rvm/gems/ruby-3.1.2/gems/mass-client-1.0.23/lib/base-line/job.rb:39:in `block in upsert'
/usr/local/rvm/gems/ruby-3.1.2/gems/mass-client-1.0.23/lib/base-line/job.rb:36:in `each'
/usr/local/rvm/gems/ruby-3.1.2/gems/mass-client-1.0.23/lib/base-line/job.rb:36:in `upsert'
/home/blackstack/code1/sdk/p/profile.rb:803:in `block in 
'
/home/blackstack/code1/sdk/p/profile.rb:764:in `each'
/home/blackstack/code1/sdk/p/profile.rb:764:in `
'

leandrosardi commented 4 weeks ago

If I kill the processes and restart them again, the lockfile keeps locked.

Timeout while waiting for lockfile.
/usr/local/rvm/gems/ruby-3.1.2/gems/mass-client-1.0.25/lib/base-line/profile.rb:21:in `rescue in acquire_lock'
/usr/local/rvm/gems/ruby-3.1.2/gems/mass-client-1.0.25/lib/base-line/profile.rb:10:in `acquire_lock'
/usr/local/rvm/gems/ruby-3.1.2/gems/mass-client-1.0.25/lib/base-line/profile.rb:31:in `upload_to_dropbox_with_lock'
/usr/local/rvm/gems/ruby-3.1.2/gems/mass-client-1.0.25/lib/first-line/profile_rpa.rb:289:in `snapshot'
/home/blackstack/code1/sdk/lib/source_types/LinkedIn_PublicFeed.rb:127:in `do'
/home/blackstack/code1/sdk/p/profile.rb:772:in `block in 
'
/home/blackstack/code1/sdk/p/profile.rb:764:in `each'
/home/blackstack/code1/sdk/p/profile.rb:764:in `
'

leandrosardi commented 3 weeks ago

I started getting error "Timeout while waiting for lockfile." very often.

leandrosardi commented 3 weeks ago

Solution

:heavy_check_mark: I learned AWS S3: https://github.com/leandrosardi/learning-aws-s3
:heavy_check_mark: Research all the places where Dropbox must be replaced by AWS.
:heavy_check_mark: Decide which part of codes to migrate at this stage ( :heavy_check_mark: ) and which parts of code ignore by now ( :x: )
:heavy_check_mark: Coding.
Testing.

leandrosardi commented 3 weeks ago

Research all the places where Dropbox must be replaced by AWS.

:x: my.saas
- :x: config.rb
- :x: config.template.rb
- :x: method def store
- :x: method def self.backup
- :x: method def self.restore
:heavy_check_mark: sdk
- :heavy_check_mark: config.rb
- :heavy_check_mark: config.template.rb
:heavy_check_mark: mass-client
- :heavy_check_mark: method upload_to_dropbox_with_lock --> upload_file_to_s3, create_s3_folder
- :heavy_check_mark: method download_image_0
- :heavy_check_mark: method wait_for_dropbox_url
- :heavy_check_mark: method screenshot
- :heavy_check_mark: method snapshot
:x: blackstack-core
- :x: method zyte_snapshot
:x: adspower-client
- :x: config.template.rb
- :x: headless.rb

MassProspecting / docs

glitch: dropbox-client error `path/not_found` --> Migrate from DropBox to AWS S3 #320

After Latest Patches

Another Problem

Solution