Closed epinault closed 12 years ago
Ruby 1.9 is supported - but using the old FasterCSV instead of Ruby 1.9 built-in CSV is not supported so far (because this case never happened).
Can you switch to the new Ruby 1.9 built-in CSV instead?
ok, so there is a dependency issue somewhere:
My Gemfile does not include it yet it comes
My Gemfile
source :gemcutter
gem "activerecord" gem "activewarehouse-etl" gem "postgres" gem "mysql" gem "foreigner"
Output:
Installing rake (0.9.2.2) Installing i18n (0.6.1) Installing multi_json (1.3.6) Installing activesupport (3.2.8) Installing builder (3.0.3) Installing activemodel (3.2.8) Installing arel (3.0.2) Installing tzinfo (0.3.33) Installing activerecord (3.2.8) Installing adapter_extensions (1.0.0) Installing fastercsv (1.5.5) Installing activewarehouse-etl (1.0.0) Installing foreigner (1.2.1) Installing mysql (2.8.1) with native extensions Installing pg (0.14.1) with native extensions Installing postgres (0.8.1) Using bundler (1.2.1)
So what should I do? remove the gem ?
We need to figure out why FasterCSV is pulled as a dependency - this may be in a dependency or in activewarehouse-etl itself. Let me run a test locally to figure this out, I'll get back to you.
Not commenting on your other issues now, let's first fix this one.
Maybe a if condition in there to add it based on RUBY_VERSION? Also wonder if a switch case in the code where you do a require also would help?
Oh I figured it out I think. You are apparently using activewarehouse-etl-0.9.1
but this does not support Ruby 1.9 yet.
Is that an old project or a new one?
Can you update to the latest (1.0.0 from june 2012), which has proper support for mysql2, ruby 1.9 etc?
I suspect most of your other issues will go away with it.
Note that I need to go offline for a bit after, but ping me if you have more issues, we'll make this work!
Hum - I may be mistaken; your stack trace mentions 0.9.1, while your gemfile refers to 1.0.0.
Could it be that you have an older version installed in parallel?
Don t know how the older one got in when trying 1.8.7 see if that would work better... . strange. But I cannot get passed anymore the following error unless I hack the code a bit. did you get a chance to look at them?
Source: localhost/ts_development/teams
/home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activerecord-3.2.8/lib/active_record/dynamic_matchers.rb:50:in `method_missing': undefined method `mysql_connection' for #<Class:0xa067ed0> (NoMethodError)
from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activewarehouse-etl-1.0.0/lib/etl/engine.rb:212:in `establish_connection'
from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activewarehouse-etl-1.0.0/lib/etl/engine.rb:143:in `connection'
from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activewarehouse-etl-1.0.0/lib/etl/control/source/database_source.rb:219:in `connection'
from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activewarehouse-etl-1.0.0/lib/etl/control/source/database_source.rb:213:in `query_rows'
from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activewarehouse-etl-1.0.0/lib/etl/control/source/database_source.rb:100:in `columns'
from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activewarehouse-etl-1.0.0/lib/etl/control/source/database_source.rb:160:in `block (2 levels) in write_local'
from /home/emmanuel/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/csv.rb:1354:in `open'
from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activewarehouse-etl-1.0.0/lib/etl/control/source/database_source.rb:159:in `block in write_local'
from /home/emmanuel/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/benchmark.rb:295:in `realtime'
from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activewarehouse-etl-1.0.0/lib/etl/control/source/database_source.rb:158:in `write_local'
from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activewarehouse-etl-1.0.0/lib/etl/control/source/database_source.rb:115:in `each'
from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activewarehouse-etl-1.0.0/lib/etl/engine.rb:333:in `each_with_index'
from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activewarehouse-etl-1.0.0/lib/etl/engine.rb:333:in `block in process_control'
from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activewarehouse-etl-1.0.0/lib/etl/engine.rb:327:in `each'
from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activewarehouse-etl-1.0.0/lib/etl/engine.rb:327:in `process_control'
from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activewarehouse-etl-1.0.0/lib/etl/engine.rb:275:in `process'
from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activewarehouse-etl-1.0.0/lib/etl/engine.rb:272:in `process'
from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activewarehouse-etl-1.0.0/lib/etl/engine.rb:55:in `process'
from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activewarehouse-etl-1.0.0/lib/etl/commands/etl.rb:82:in `block in execute'
from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activewarehouse-etl-1.0.0/lib/etl/commands/etl.rb:80:in `each'
from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activewarehouse-etl-1.0.0/lib/etl/commands/etl.rb:80:in `execute'
from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activewarehouse-etl-1.0.0/lib/etl/commands/etl.rb:90:in `<top (required)>'
from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activesupport-3.2.8/lib/active_support/dependencies.rb:251:in `require'
from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activesupport-3.2.8/lib/active_support/dependencies.rb:251:in `block in require'
from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activesupport-3.2.8/lib/active_support/dependencies.rb:236:in `load_dependency'
from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activesupport-3.2.8/lib/active_support/dependencies.rb:251:in `require'
from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activewarehouse-etl-1.0.0/bin/etl:28:in `<top (required)>'
from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/bin/etl:23:in `load'
from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/bin/etl:23:in `<main>'
This is actually odd since bundler says it was also 1.0 installed too !
I cannot reproduce it with 1.9.2 so this is a more of a 1.8 issue. Can you help me with the missing method? this is the one really blocking me
Most likely the previous version was installed without bundler, so it is accessible in the command line first! But this one is done.
I'm now having a look at the missing method issue, hold tight :)
I think you are also using an old version of mysql gem, for some reason!
Make sure:
mysql2
and not mysql
in your Gemfileadapter: mysql2
in your database.ymlAnd try again - if you have any more issue, let me know!
I tried both and got the same results :(
Gems included by the bundle:
Database.yml
common_mysql: &common_mysql adapter: mysql2 username: root host: localhost
common_postgre: &common_postgre adapter: postgresql username: etl host: localhost
etl_execution: <<: *common_postgre database: etl_execution
talemetry_match_web: <<: *common_mysql database: ts_development encoding: utf8
talemetry_match_crawl: <<: *common_mysql database: ts_web_production encoding: utf8
datawarehouse: <<: *common_postgre database: talemetry_bi encoding: utf8
and error
home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activerecord-3.2.8/lib/active_record/dynamic_matchers.rb:50:in `method_missing': undefined method `mysql2_connection' for #<Class:0xa79fb44> (NoMethodError)
from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activewarehouse-etl-1.0.0/lib/etl/engine.rb:212:in `establish_connection'
from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activewarehouse-etl-1.0.0/lib/etl/engine.rb:143:in `connection'
from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activewarehouse-etl-1.0.0/lib/etl/control/source/database_source.rb:219:in `connection'
from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activewarehouse-etl-1.0.0/lib/etl/control/source/database_source.rb:213:in `query_rows'
from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activewarehouse-etl-1.0.0/lib/etl/control/source/database_source.rb:100:in `columns'
from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activewarehouse-etl-1.0.0/lib/etl/control/source/database_source.rb:160:in `block (2 levels) in write_local'
from /home/emmanuel/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/csv.rb:1354:in `open'
from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activewarehouse-etl-1.0.0/lib/etl/control/source/database_source.rb:159:in `block in write_local'
from /home/emmanuel/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/benchmark.rb:295:in `realtime'
from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activewarehouse-etl-1.0.0/lib/etl/control/source/database_source.rb:158:in `write_local'
from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activewarehouse-etl-1.0.0/lib/etl/control/source/database_source.rb:115:in `each'
from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activewarehouse-etl-1.0.0/lib/etl/engine.rb:333:in `each_with_index'
from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activewarehouse-etl-1.0.0/lib/etl/engine.rb:333:in `block in process_control'
from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activewarehouse-etl-1.0.0/lib/etl/engine.rb:327:in `each'
from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activewarehouse-etl-1.0.0/lib/etl/engine.rb:327:in `process_control'
from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activewarehouse-etl-1.0.0/lib/etl/engine.rb:275:in `process'
from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activewarehouse-etl-1.0.0/lib/etl/engine.rb:272:in `process'
from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activewarehouse-etl-1.0.0/lib/etl/engine.rb:55:in `process'
from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activewarehouse-etl-1.0.0/lib/etl/commands/etl.rb:82:in `block in execute'
from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activewarehouse-etl-1.0.0/lib/etl/commands/etl.rb:80:in `each'
from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activewarehouse-etl-1.0.0/lib/etl/commands/etl.rb:80:in `execute'
from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activewarehouse-etl-1.0.0/lib/etl/commands/etl.rb:90:in `<top (required)>'
from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activesupport-3.2.8/lib/active_support/dependencies.rb:251:in `require'
from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activesupport-3.2.8/lib/active_support/dependencies.rb:251:in `block in require'
from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activesupport-3.2.8/lib/active_support/dependencies.rb:236:in `load_dependency'
from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activesupport-3.2.8/lib/active_support/dependencies.rb:251:in `require'
from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activewarehouse-etl-1.0.0/bin/etl:28:in `<top (required)>'
from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/bin/etl:23:in `load'
Also I am not using Rails directly and trying configure it with the database.yml per your doc. I see the postgresql_connection method in the ETL::Base but nothing around mysql. Something is not loading Mysql in AR correctly?
I'm short on time right now (will get back to this tonight in more depth), but yes it seems there is an issue with loading the MySQL AR or something.
Please comment here if you find something until I have a closer look tonight!
Sounds good. I ll be on till 11pm France time ;) So we can work on it together if you want
Can you invite me to google chat at thibaut.barrere@gmail.com please so we can discuss that?
Currently I believe the issue is linked to the fact that you use both pg and mysql in the same setup.
I will probably need to have a closer look at your ETL scripts privately to figure things out here.
sure here is what it does so far :)
class Account < ActiveRecord::Base end
source :input, { :type => :database, :target => :talemetry_match_web, :table => "teams" }, [ :name, :company_id, :talemetry_id, :country_code, :status, :ofccp, ]
transform(:name) do |n,v,r| p v p n p r r[:name] end
destination :out, { :type => :database, :target => :datawarehouse, :table => 'accounts' }, { :primarykey => [:email], :order => [:email, :full_name, :email_provider] }
If going to DB(mysql) => File => File => DB (postgre) works then I might take that route
I tried just mysql to file and not working either with same error...
Try requiring 'mysql2' (if you have 'mysql2' in your database.yml) at the top of you control file maybe?
I won't be able to help much more tonight. If this doesn't work contact me by email so we can discuss that tomorrow afternoon hopefully.
will try that and report back. Is your email posted on here somewhere?
I put it on top of my ctl file and not working
gonna try this http://stackoverflow.com/questions/1298909/multiple-database-connection-in-rails
My email is on my github profile: https://github.com/thbar
This looks a bit complicated - keep me posted in case you find something!
here is what I found
Adding the following to the top of my ctl file
$config = YAML.load_file(File.join(File.dirname(FILE), '../config/mysql.yml'))
class Account < ActiveRecord::Base establish_connection $config['talemetry_match_web'] end
where th mysql.yml is only setup for mysql makes it go further though it breaks with
home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activerecord-mysql2-adapter-0.0.3/lib/active_record/connection_adapters/mysql2_adapter.rb:614:in select': wrong number of arguments (3 for 2) (ArgumentError) from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activerecord-3.2.8/lib/active_record/connection_adapters/abstract/database_statements.rb:18:in
select_all'
from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activerecord-3.2.8/lib/active_record/connection_adapters/abstract/query_cache.rb:63:in select_all' from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activewarehouse-etl-1.0.0/lib/etl/control/source/database_source.rb:213:in
query_rows'
from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activewarehouse-etl-1.0.0/lib/etl/control/source/database_source.rb:100:in columns' from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activewarehouse-etl-1.0.0/lib/etl/control/source/database_source.rb:160:in
block (2 levels) in write_local'
from /home/emmanuel/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/csv.rb:1354:in open' from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activewarehouse-etl-1.0.0/lib/etl/control/source/database_source.rb:159:in
block in write_local'
from /home/emmanuel/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/benchmark.rb:295:in realtime' from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activewarehouse-etl-1.0.0/lib/etl/control/source/database_source.rb:158:in
write_local'
from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activewarehouse-etl-1.0.0/lib/etl/control/source/database_source.rb:115:in each' from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activewarehouse-etl-1.0.0/lib/etl/engine.rb:334:in
each_with_index'
from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activewarehouse-etl-1.0.0/lib/etl/engine.rb:334:in block in process_control' from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activewarehouse-etl-1.0.0/lib/etl/engine.rb:328:in
each'
from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activewarehouse-etl-1.0.0/lib/etl/engine.rb:328:in process_control' from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activewarehouse-etl-1.0.0/lib/etl/engine.rb:276:in
process'
from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activewarehouse-etl-1.0.0/lib/etl/engine.rb:273:in process' from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activewarehouse-etl-1.0.0/lib/etl/engine.rb:55:in
process'
from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activewarehouse-etl-1.0.0/lib/etl/commands/etl.rb:82:in block in execute' from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activewarehouse-etl-1.0.0/lib/etl/commands/etl.rb:80:in
each'
from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activewarehouse-etl-1.0.0/lib/etl/commands/etl.rb:80:in execute' from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activewarehouse-etl-1.0.0/lib/etl/commands/etl.rb:90:in
<top (required)>'
from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activesupport-3.2.8/lib/active_support/dependencies.rb:251:in require' from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activesupport-3.2.8/lib/active_support/dependencies.rb:251:in
block in require'
from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activesupport-3.2.8/lib/active_support/dependencies.rb:236:in load_dependency' from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activesupport-3.2.8/lib/active_support/dependencies.rb:251:in
require'
from /home/emmanuel/talemetry/talemetry_warehouse/vendor/ruby/1.9.1/gems/activewarehouse-etl-1.0.0/bin/etl:28:in `<top (required)>'
fr
So the original error (FasterCSV) is solved by upgrading to the latest version of activewarehouse-etl.
I'm closing this one - the end of the issue will be tracked on #110.
I am getting the following error when trying to run an ETL ctl. Is 1.9.x supported? how do I fix this?