Import data from MS SQL Server too slowly

[fengme]

We can only import not more than 2 million records with 8 hours, there are almost 40 million records in all, I wonder how to improve the importing efficiency, thanks!

The following is the Logstash configuration file: input { jdbc { jdbc_validate_connection => true jdbc_connection_string => "jdbc:sqlserver://IP:1433;databasename=RoadCenter;user=sa,password=PWD" jdbc_user => "sa" jdbc_password => "PWD" jdbc_driver_library => "/opt/es/mssql-jdbc-6.2.2.jre8.jar" jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver" jdbc_paging_enabled => "true" jdbc_page_size => "50000" statement => "SELECT FROM O_TRANSACTION" schedule => "/1 " type => "jdbc_mssql" } }

filter { ruby { code => 'event.set("time_take",event.get("dhm").to_i-event.get("edtime").to_i)' } mutate { remove_field => "message" } }

output {
elasticsearch { hosts => ["http://192.168.1.205:9200"] index => "mssql_o_transaction_3"
workers => 1 }

stdout {codec => rubydebug}

}

[guyboertje]

@fengme

Are you still experiencing this? Did you make progress?