OrderByOperator extremely overestimates the memory usage

alexey-fin commented 4 years ago

@findepi query "select * from testorc order by random()" estimates memory to more than 50gb while heap does not cross 10gb, orc file https://drive.google.com/file/d/1DD7Kxcy3CodKaIDPP-G26n1PVWT20fTh/view?usp=sharing, create table : CREATE TABLE hive.default.testorc ( salesforce_contact_id varchar, email_id varchar, campaign_id varchar, sfmc_activity_id varchar, campaign_name varchar, sfmc_activity_type varchar, sfmc_activity_sub_type varchar, day date, is_delivered double, is_open double, is_inferred_open double, total_opens double, is_link_clicked double, is_hard_bounce double, is_bounce double, is_opt_out double, is_sent double, gl_lob varchar, region varchar, subregion varchar, company_id_l1 varchar, tier varchar, sales_channel varchar, channel_name varchar, primary_topics varchar, secondary_topics varchar, subs_frequency varchar(13), url_clicked_personalized varchar, url_alias_80 varchar, territory_code_1 varchar, territory_code_2 varchar, territory_code_3 varchar, territory_code_4 varchar, territory_code_5 varchar, country_full varchar, email_cost decimal(2, 1), web_engaged_sub decimal(2, 1), email_engaged_sub decimal(2, 1), mps_flag decimal(2, 1), opportunities bigint, is_sub double, mpc_flag integer, downloads double, video_views double, logins double, pageviews double, guide_to_markets_interactions double, market_insights_email_signups double, registrations double, podcast_plays double, bond_ladder_tool_analysis double, portfolio_insights_tool_analysis double, investment_comparison_tool_analysis double, target_date_compass_tool_analysis double, bond_ladder_tool_report double, portfolio_insights_tool_report double, investment_comparison_tool_report double, target_date_compass_tool_report double, total_tool_report double, total_tool_analysis double, total double ) WITH ( external_location = 'file:/Users/afinkelstein/testutf/testorc', format = 'ORC' )

tooptoop4 commented 4 years ago

see if https://github.com/prestosql/presto/pull/2612 helps

alexey-fin commented 4 years ago

@tooptoop4 i tried pageIndex compact, i will try the page compact as well

alexey-fin commented 4 years ago

It's in master so yeah, didn't help unfortunately

sopel39 commented 4 years ago

Could you share query JSON? Do you use Presto version that has https://github.com/prestosql/presto/pull/2612 inside?

kfirt commented 4 years ago

we are using version 326.

alexey-fin commented 4 years ago

@sopel39 yes #2612 is inside, what do you mean query JSON sorry?

tooptoop4 commented 4 years ago

on webui click your queryid then on far right json tab

alexey-fin commented 4 years ago

that's one from a failed run: { "queryId" : "20200316_143152_00013_fbfw4", "session" : { "queryId" : "20200316_143152_00013_fbfw4", "transactionId" : "ddbaac3f-cd10-4a44-bf9a-8bdbff520516", "clientTransactionSupport" : true, "user" : "test", "source" : "presto-jdbc", "catalog" : "hive", "schema" : "default", "path" : { }, "timeZoneKey" : 1946, "locale" : "en_IL", "remoteUserAddress" : "127.0.0.1", "userAgent" : "Presto JDBC Driver/328", "clientTags" : [ ], "clientCapabilities" : [ "PATH" ], "resourceEstimates" : { }, "startTime" : 1584369112861, "systemProperties" : { }, "catalogProperties" : { }, "unprocessedCatalogProperties" : { }, "roles" : { }, "preparedStatements" : { } }, "state" : "FAILED", "memoryPool" : "general", "scheduled" : true, "self" : "http://localhost:8080/v1/query/20200316_143152_00013_fbfw4", "fieldNames" : [ "salesforce_contact_id", "email_id", "campaign_id", "sfmc_activity_id", "campaign_name", "sfmc_activity_type", "sfmc_activity_sub_type", "day", "is_delivered", "is_open", "is_inferred_open", "total_opens", "is_link_clicked", "is_hard_bounce", "is_bounce", "is_opt_out", "is_sent", "gl_lob", "region", "subregion", "company_id_l1", "tier", "sales_channel", "channel_name", "primary_topics", "secondary_topics", "subs_frequency", "url_clicked_personalized", "url_alias_80", "territory_code_1", "territory_code_2", "territory_code_3", "territory_code_4", "territory_code_5", "country_full", "email_cost", "web_engaged_sub", "email_engaged_sub", "mps_flag", "opportunities", "is_sub", "mpc_flag", "downloads", "video_views", "logins", "pageviews", "guide_to_markets_interactions", "market_insights_email_signups", "registrations", "podcast_plays", "bond_ladder_tool_analysis", "portfolio_insights_tool_analysis", "investment_comparison_tool_analysis", "target_date_compass_tool_analysis", "bond_ladder_tool_report", "portfolio_insights_tool_report", "investment_comparison_tool_report", "target_date_compass_tool_report", "total_tool_report", "total_tool_analysis", "total" ], "query" : "select * from testorc order by random()", "queryStats" : { "createTime" : "2020-03-16T14:31:52.871Z", "executionStartTime" : "2020-03-16T14:31:52.958Z", "lastHeartbeat" : "2020-03-16T14:32:23.709Z", "endTime" : "2020-03-16T14:32:24.242Z", "elapsedTime" : "31.37s", "queuedTime" : "660.88us", "resourceWaitingTime" : "87.19ms", "dispatchingTime" : "145.06us", "executionTime" : "31.28s", "analysisTime" : "86.98ms", "planningTime" : "197.40ms", "finishingTime" : "0.00ns", "totalTasks" : 3, "runningTasks" : 0, "completedTasks" : 3, "totalDrivers" : 24, "queuedDrivers" : 0, "runningDrivers" : 1, "blockedDrivers" : 18, "completedDrivers" : 5, "cumulativeUserMemory" : 7.46334276797507E14, "userMemoryReservation" : "53678376821B", "revocableMemoryReservation" : "0B", "totalMemoryReservation" : "53697314134B", "peakUserMemoryReservation" : "52565356430B", "peakRevocableMemoryReservation" : "0B", "peakTotalMemoryReservation" : "52632910684B", "peakTaskUserMemory" : "52565356430B", "peakTaskRevocableMemory" : "0B", "peakTaskTotalMemory" : "52565356430B", "scheduled" : true, "totalScheduledTime" : "3.18m", "totalCpuTime" : "1.19m", "totalBlockedTime" : "8.64m", "fullyBlocked" : true, "blockedReasons" : [ ], "physicalInputDataSize" : "181576997B", "physicalInputPositions" : 5198526, "physicalInputReadTime" : "1.10s", "internalNetworkInputDataSize" : "2110724193B", "internalNetworkInputPositions" : 5197502, "rawInputDataSize" : "181576997B", "rawInputPositions" : 5198526, "processedInputDataSize" : "2327849012B", "processedInputPositions" : 5198526, "outputDataSize" : "0B", "outputPositions" : 0, "physicalWrittenDataSize" : "0B", "stageGcStatistics" : [ { "stageId" : 0, "tasks" : 1, "fullGcTasks" : 0, "minFullGcSec" : 0, "maxFullGcSec" : 0, "totalFullGcSec" : 0, "averageFullGcSec" : 0 }, { "stageId" : 1, "tasks" : 1, "fullGcTasks" : 0, "minFullGcSec" : 0, "maxFullGcSec" : 0, "totalFullGcSec" : 0, "averageFullGcSec" : 0 }, { "stageId" : 2, "tasks" : 1, "fullGcTasks" : 0, "minFullGcSec" : 0, "maxFullGcSec" : 0, "totalFullGcSec" : 0, "averageFullGcSec" : 0 } ], "operatorSummaries" : [ { "stageId" : 0, "pipelineId" : 0, "operatorId" : 0, "planNodeId" : "166", "operatorType" : "MergeOperator", "totalDrivers" : 1, "addInputCalls" : 0, "addInputWall" : "0.00ns", "addInputCpu" : "0.00ns", "physicalInputDataSize" : "0B", "physicalInputPositions" : 0, "internalNetworkInputDataSize" : "0B", "internalNetworkInputPositions" : 0, "rawInputDataSize" : "0B", "inputDataSize" : "0B", "inputPositions" : 0, "sumSquaredInputPositions" : 0.0, "getOutputCalls" : 1, "getOutputWall" : "122.66us", "getOutputCpu" : "115.00us", "outputDataSize" : "0B", "outputPositions" : 0, "physicalWrittenDataSize" : "0B", "blockedWall" : "0.00ns", "finishCalls" : 0, "finishWall" : "0.00ns", "finishCpu" : "0.00ns", "userMemoryReservation" : "0B", "revocableMemoryReservation" : "0B", "systemMemoryReservation" : "0B", "peakUserMemoryReservation" : "0B", "peakSystemMemoryReservation" : "0B", "peakRevocableMemoryReservation" : "0B", "peakTotalMemoryReservation" : "0B", "spilledDataSize" : "0B" }, { "stageId" : 0, "pipelineId" : 0, "operatorId" : 1, "planNodeId" : "4", "operatorType" : "FilterAndProjectOperator", "totalDrivers" : 1, "addInputCalls" : 0, "addInputWall" : "0.00ns", "addInputCpu" : "0.00ns", "physicalInputDataSize" : "0B", "physicalInputPositions" : 0, "internalNetworkInputDataSize" : "0B", "internalNetworkInputPositions" : 0, "rawInputDataSize" : "0B", "inputDataSize" : "0B", "inputPositions" : 0, "sumSquaredInputPositions" : 0.0, "getOutputCalls" : 2, "getOutputWall" : "80.29us", "getOutputCpu" : "83.00us", "outputDataSize" : "0B", "outputPositions" : 0, "physicalWrittenDataSize" : "0B", "blockedWall" : "0.00ns", "finishCalls" : 0, "finishWall" : "0.00ns", "finishCpu" : "0.00ns", "userMemoryReservation" : "0B", "revocableMemoryReservation" : "0B", "systemMemoryReservation" : "0B", "peakUserMemoryReservation" : "0B", "peakSystemMemoryReservation" : "0B", "peakRevocableMemoryReservation" : "0B", "peakTotalMemoryReservation" : "0B", "spilledDataSize" : "0B" }, { "stageId" : 0, "pipelineId" : 0, "operatorId" : 2, "planNodeId" : "7", "operatorType" : "TaskOutputOperator", "totalDrivers" : 1, "addInputCalls" : 0, "addInputWall" : "0.00ns", "addInputCpu" : "0.00ns", "physicalInputDataSize" : "0B", "physicalInputPositions" : 0, "internalNetworkInputDataSize" : "0B", "internalNetworkInputPositions" : 0, "rawInputDataSize" : "0B", "inputDataSize" : "0B", "inputPositions" : 0, "sumSquaredInputPositions" : 0.0, "getOutputCalls" : 0, "getOutputWall" : "0.00ns", "getOutputCpu" : "0.00ns", "outputDataSize" : "0B", "outputPositions" : 0, "physicalWrittenDataSize" : "0B", "blockedWall" : "0.00ns", "finishCalls" : 0, "finishWall" : "0.00ns", "finishCpu" : "0.00ns", "userMemoryReservation" : "0B", "revocableMemoryReservation" : "0B", "systemMemoryReservation" : "0B", "peakUserMemoryReservation" : "0B", "peakSystemMemoryReservation" : "0B", "peakRevocableMemoryReservation" : "0B", "peakTotalMemoryReservation" : "0B", "spilledDataSize" : "0B" }, { "stageId" : 1, "pipelineId" : 0, "operatorId" : 0, "planNodeId" : "165", "operatorType" : "ExchangeOperator", "totalDrivers" : 16, "addInputCalls" : 0, "addInputWall" : "0.00ns", "addInputCpu" : "0.00ns", "physicalInputDataSize" : "0B", "physicalInputPositions" : 0, "internalNetworkInputDataSize" : "2110724193B", "internalNetworkInputPositions" : 5197502, "rawInputDataSize" : "2110724193B", "inputDataSize" : "2374995020B", "inputPositions" : 5197502, "sumSquaredInputPositions" : 1.695206720428E12, "getOutputCalls" : 5308, "getOutputWall" : "42.85s", "getOutputCpu" : "11.51s", "outputDataSize" : "2374995020B", "outputPositions" : 5197502, "physicalWrittenDataSize" : "0B", "blockedWall" : "7.22m", "finishCalls" : 0, "finishWall" : "0.00ns", "finishCpu" : "0.00ns", "userMemoryReservation" : "0B", "revocableMemoryReservation" : "0B", "systemMemoryReservation" : "0B", "peakUserMemoryReservation" : "0B", "peakSystemMemoryReservation" : "0B", "peakRevocableMemoryReservation" : "0B", "peakTotalMemoryReservation" : "0B", "spilledDataSize" : "0B", "info" : { "@type" : "exchangeClientStatus", "bufferedBytes" : 0, "maxBufferedBytes" : 12581962, "averageBytesPerRequest" : 5704461, "successfulRequestsCount" : 5920, "bufferedPages" : 0, "noMoreLocations" : true, "pageBufferClientStatuses" : [ ] } }, { "stageId" : 1, "pipelineId" : 1, "operatorId" : 0, "planNodeId" : "204", "operatorType" : "LocalMergeSourceOperator", "totalDrivers" : 1, "addInputCalls" : 0, "addInputWall" : "0.00ns", "addInputCpu" : "0.00ns", "physicalInputDataSize" : "0B", "physicalInputPositions" : 0, "internalNetworkInputDataSize" : "0B", "internalNetworkInputPositions" : 0, "rawInputDataSize" : "0B", "inputDataSize" : "0B", "inputPositions" : 0, "sumSquaredInputPositions" : 0.0, "getOutputCalls" : 1, "getOutputWall" : "1.67ms", "getOutputCpu" : "234.00us", "outputDataSize" : "0B", "outputPositions" : 0, "physicalWrittenDataSize" : "0B", "blockedWall" : "0.00ns", "finishCalls" : 0, "finishWall" : "0.00ns", "finishCpu" : "0.00ns", "userMemoryReservation" : "0B", "revocableMemoryReservation" : "0B", "systemMemoryReservation" : "0B", "peakUserMemoryReservation" : "0B", "peakSystemMemoryReservation" : "0B", "peakRevocableMemoryReservation" : "0B", "peakTotalMemoryReservation" : "0B", "spilledDataSize" : "0B" }, { "stageId" : 1, "pipelineId" : 0, "operatorId" : 1, "planNodeId" : "167", "operatorType" : "OrderByOperator", "totalDrivers" : 16, "addInputCalls" : 5214, "addInputWall" : "13.95s", "addInputCpu" : "4.53s", "physicalInputDataSize" : "0B", "physicalInputPositions" : 0, "internalNetworkInputDataSize" : "0B", "internalNetworkInputPositions" : 0, "rawInputDataSize" : "0B", "inputDataSize" : "2374995020B", "inputPositions" : 5197502, "sumSquaredInputPositions" : 1.695206720428E12, "getOutputCalls" : 10924, "getOutputWall" : "1.42s", "getOutputCpu" : "297.01ms", "outputDataSize" : "0B", "outputPositions" : 0, "physicalWrittenDataSize" : "0B", "blockedWall" : "0.00ns", "finishCalls" : 0, "finishWall" : "0.00ns", "finishCpu" : "0.00ns", "userMemoryReservation" : "53678376821B", "revocableMemoryReservation" : "0B", "systemMemoryReservation" : "0B", "peakUserMemoryReservation" : "3783289414B", "peakSystemMemoryReservation" : "0B", "peakRevocableMemoryReservation" : "0B", "peakTotalMemoryReservation" : "3783289414B", "spilledDataSize" : "0B" }, { "stageId" : 1, "pipelineId" : 1, "operatorId" : 1, "planNodeId" : "204", "operatorType" : "TaskOutputOperator", "totalDrivers" : 1, "addInputCalls" : 0, "addInputWall" : "0.00ns", "addInputCpu" : "0.00ns", "physicalInputDataSize" : "0B", "physicalInputPositions" : 0, "internalNetworkInputDataSize" : "0B", "internalNetworkInputPositions" : 0, "rawInputDataSize" : "0B", "inputDataSize" : "0B", "inputPositions" : 0, "sumSquaredInputPositions" : 0.0, "getOutputCalls" : 0, "getOutputWall" : "0.00ns", "getOutputCpu" : "0.00ns", "outputDataSize" : "0B", "outputPositions" : 0, "physicalWrittenDataSize" : "0B", "blockedWall" : "0.00ns", "finishCalls" : 0, "finishWall" : "0.00ns", "finishCpu" : "0.00ns", "userMemoryReservation" : "0B", "revocableMemoryReservation" : "0B", "systemMemoryReservation" : "0B", "peakUserMemoryReservation" : "0B", "peakSystemMemoryReservation" : "0B", "peakRevocableMemoryReservation" : "0B", "peakTotalMemoryReservation" : "0B", "spilledDataSize" : "0B" }, { "stageId" : 1, "pipelineId" : 0, "operatorId" : 2, "planNodeId" : "204", "operatorType" : "LocalExchangeSinkOperator", "totalDrivers" : 16, "addInputCalls" : 0, "addInputWall" : "0.00ns", "addInputCpu" : "0.00ns", "physicalInputDataSize" : "0B", "physicalInputPositions" : 0, "internalNetworkInputDataSize" : "0B", "internalNetworkInputPositions" : 0, "rawInputDataSize" : "0B", "inputDataSize" : "0B", "inputPositions" : 0, "sumSquaredInputPositions" : 0.0, "getOutputCalls" : 0, "getOutputWall" : "0.00ns", "getOutputCpu" : "0.00ns", "outputDataSize" : "0B", "outputPositions" : 0, "physicalWrittenDataSize" : "0B", "blockedWall" : "0.00ns", "finishCalls" : 0, "finishWall" : "0.00ns", "finishCpu" : "0.00ns", "userMemoryReservation" : "0B", "revocableMemoryReservation" : "0B", "systemMemoryReservation" : "0B", "peakUserMemoryReservation" : "0B", "peakSystemMemoryReservation" : "0B", "peakRevocableMemoryReservation" : "0B", "peakTotalMemoryReservation" : "0B", "spilledDataSize" : "0B" }, { "stageId" : 2, "pipelineId" : 0, "operatorId" : 0, "planNodeId" : "2", "operatorType" : "ScanFilterAndProjectOperator", "totalDrivers" : 6, "addInputCalls" : 5208, "addInputWall" : "1.10s", "addInputCpu" : "0.00ns", "physicalInputDataSize" : "181576997B", "physicalInputPositions" : 5198526, "internalNetworkInputDataSize" : "0B", "internalNetworkInputPositions" : 0, "rawInputDataSize" : "181576997B", "inputDataSize" : "2327849012B", "inputPositions" : 5198526, "sumSquaredInputPositions" : 5.188204009348E12, "getOutputCalls" : 5235, "getOutputWall" : "1.04m", "getOutputCpu" : "26.02s", "outputDataSize" : "2375477893B", "outputPositions" : 5198526, "physicalWrittenDataSize" : "0B", "blockedWall" : "0.00ns", "finishCalls" : 0, "finishWall" : "0.00ns", "finishCpu" : "0.00ns", "userMemoryReservation" : "0B", "revocableMemoryReservation" : "0B", "systemMemoryReservation" : "18937313B", "peakUserMemoryReservation" : "0B", "peakSystemMemoryReservation" : "38629863B", "peakRevocableMemoryReservation" : "0B", "peakTotalMemoryReservation" : "38629863B", "spilledDataSize" : "0B" }, { "stageId" : 2, "pipelineId" : 0, "operatorId" : 1, "planNodeId" : "2", "operatorType" : "TaskOutputOperator", "totalDrivers" : 6, "addInputCalls" : 5214, "addInputWall" : "1.16m", "addInputCpu" : "28.98s", "physicalInputDataSize" : "0B", "physicalInputPositions" : 0, "internalNetworkInputDataSize" : "0B", "internalNetworkInputPositions" : 0, "rawInputDataSize" : "0B", "inputDataSize" : "2374995020B", "inputPositions" : 5197502, "sumSquaredInputPositions" : 5.18595134042E12, "getOutputCalls" : 0, "getOutputWall" : "0.00ns", "getOutputCpu" : "0.00ns", "outputDataSize" : "2374995020B", "outputPositions" : 5197502, "physicalWrittenDataSize" : "0B", "blockedWall" : "22.17s", "finishCalls" : 5, "finishWall" : "74.68us", "finishCpu" : "98.00us", "userMemoryReservation" : "0B", "revocableMemoryReservation" : "0B", "systemMemoryReservation" : "0B", "peakUserMemoryReservation" : "0B", "peakSystemMemoryReservation" : "0B", "peakRevocableMemoryReservation" : "0B", "peakTotalMemoryReservation" : "0B", "spilledDataSize" : "0B" } ], "logicalWrittenDataSize" : "0B", "writtenPositions" : 0, "spilledDataSize" : "0B", "progressPercentage" : 20.833333333333332 }, "setSessionProperties" : { }, "resetSessionProperties" : [ ], "setRoles" : { }, "addedPreparedStatements" : { }, "deallocatedPreparedStatements" : [ ], "clearTransactionId" : false, "outputStage" : { "stageId" : "20200316_143152_00013_fbfw4.0", "state" : "ABORTED", "plan" : { "id" : "0", "root" : { "@type" : "output", "id" : "7", "source" : { "@type" : "project", "id" : "4", "source" : { "@type" : "remoteSource", "id" : "166", "sourceFragmentIds" : [ "1" ], "outputs" : [ "random", "salesforce_contact_id", "email_id", "campaign_id", "sfmc_activity_id", "campaign_name", "sfmc_activity_type", "sfmc_activity_sub_type", "day", "is_delivered", "is_open", "is_inferred_open", "total_opens", "is_link_clicked", "is_hard_bounce", "is_bounce", "is_opt_out", "is_sent", "gl_lob", "region", "subregion", "company_id_l1", "tier", "sales_channel", "channel_name", "primary_topics", "secondary_topics", "subs_frequency", "url_clicked_personalized", "url_alias", "territory_code", "territory_code_0", "territory_code_1", "territory_code_2", "territory_code_3", "country_full", "email_cost", "web_engaged_sub", "email_engaged_sub", "mps_flag", "opportunities", "is_sub", "mpc_flag", "downloads", "video_views", "logins", "pageviews", "guide_to_markets_interactions", "market_insights_email_signups", "registrations", "podcast_plays", "bond_ladder_tool_analysis", "portfolio_insights_tool_analysis", "investment_comparison_tool_analysis", "target_date_compass_tool_analysis", "bond_ladder_tool_report", "portfolio_insights_tool_report", "investment_comparison_tool_report", "target_date_compass_tool_report", "total_tool_report", "total_tool_analysis", "total" ], "orderingScheme" : { "orderBy" : [ "random" ], "orderings" : { "random" : "ASC_NULLS_LAST" } }, "exchangeType" : "GATHER" }, "assignments" : { "assignments" : { "salesforce_contact_id" : "\"salesforce_contact_id\"", "email_id" : "\"email_id\"", "campaign_id" : "\"campaign_id\"", "sfmc_activity_id" : "\"sfmc_activity_id\"", "campaign_name" : "\"campaign_name\"", "sfmc_activity_type" : "\"sfmc_activity_type\"", "sfmc_activity_sub_type" : "\"sfmc_activity_sub_type\"", "day" : "\"day\"", "is_delivered" : "\"is_delivered\"", "is_open" : "\"is_open\"", "is_inferred_open" : "\"is_inferred_open\"", "total_opens" : "\"total_opens\"", "is_link_clicked" : "\"is_link_clicked\"", "is_hard_bounce" : "\"is_hard_bounce\"", "is_bounce" : "\"is_bounce\"", "is_opt_out" : "\"is_opt_out\"", "is_sent" : "\"is_sent\"", "gl_lob" : "\"gl_lob\"", "region" : "\"region\"", "subregion" : "\"subregion\"", "company_id_l1" : "\"company_id_l1\"", "tier" : "\"tier\"", "sales_channel" : "\"sales_channel\"", "channel_name" : "\"channel_name\"", "primary_topics" : "\"primary_topics\"", "secondary_topics" : "\"secondary_topics\"", "subs_frequency" : "\"subs_frequency\"", "url_clicked_personalized" : "\"url_clicked_personalized\"", "url_alias" : "\"url_alias\"", "territory_code" : "\"territory_code\"", "territory_code_0" : "\"territory_code_0\"", "territory_code_1" : "\"territory_code_1\"", "territory_code_2" : "\"territory_code_2\"", "territory_code_3" : "\"territory_code_3\"", "country_full" : "\"country_full\"", "email_cost" : "\"email_cost\"", "web_engaged_sub" : "\"web_engaged_sub\"", "email_engaged_sub" : "\"email_engaged_sub\"", "mps_flag" : "\"mps_flag\"", "opportunities" : "\"opportunities\"", "is_sub" : "\"is_sub\"", "mpc_flag" : "\"mpc_flag\"", "downloads" : "\"downloads\"", "video_views" : "\"video_views\"", "logins" : "\"logins\"", "pageviews" : "\"pageviews\"", "guide_to_markets_interactions" : "\"guide_to_markets_interactions\"", "market_insights_email_signups" : "\"market_insights_email_signups\"", "registrations" : "\"registrations\"", "podcast_plays" : "\"podcast_plays\"", "bond_ladder_tool_analysis" : "\"bond_ladder_tool_analysis\"", "portfolio_insights_tool_analysis" : "\"portfolio_insights_tool_analysis\"", "investment_comparison_tool_analysis" : "\"investment_comparison_tool_analysis\"", "target_date_compass_tool_analysis" : "\"target_date_compass_tool_analysis\"", "bond_ladder_tool_report" : "\"bond_ladder_tool_report\"", "portfolio_insights_tool_report" : "\"portfolio_insights_tool_report\"", "investment_comparison_tool_report" : "\"investment_comparison_tool_report\"", "target_date_compass_tool_report" : "\"target_date_compass_tool_report\"", "total_tool_report" : "\"total_tool_report\"", "total_tool_analysis" : "\"total_tool_analysis\"", "total" : "\"total\"" } } }, "columns" : [ "salesforce_contact_id", "email_id", "campaign_id", "sfmc_activity_id", "campaign_name", "sfmc_activity_type", "sfmc_activity_sub_type", "day", "is_delivered", "is_open", "is_inferred_open", "total_opens", "is_link_clicked", "is_hard_bounce", "is_bounce", "is_opt_out", "is_sent", "gl_lob", "region", "subregion", "company_id_l1", "tier", "sales_channel", "channel_name", "primary_topics", "secondary_topics", "subs_frequency", "url_clicked_personalized", "url_alias_80", "territory_code_1", "territory_code_2", "territory_code_3", "territory_code_4", "territory_code_5", "country_full", "email_cost", "web_engaged_sub", "email_engaged_sub", "mps_flag", "opportunities", "is_sub", "mpc_flag", "downloads", "video_views", "logins", "pageviews", "guide_to_markets_interactions", "market_insights_email_signups", "registrations", "podcast_plays", "bond_ladder_tool_analysis", "portfolio_insights_tool_analysis", "investment_comparison_tool_analysis", "target_date_compass_tool_analysis", "bond_ladder_tool_report", "portfolio_insights_tool_report", "investment_comparison_tool_report", "target_date_compass_tool_report", "total_tool_report", "total_tool_analysis", "total" ], "outputs" : [ "salesforce_contact_id", "email_id", "campaign_id", "sfmc_activity_id", "campaign_name", "sfmc_activity_type", "sfmc_activity_sub_type", "day", "is_delivered", "is_open", "is_inferred_open", "total_opens", "is_link_clicked", "is_hard_bounce", "is_bounce", "is_opt_out", "is_sent", "gl_lob", "region", "subregion", "company_id_l1", "tier", "sales_channel", "channel_name", "primary_topics", "secondary_topics", "subs_frequency", "url_clicked_personalized", "url_alias", "territory_code", "territory_code_0", "territory_code_1", "territory_code_2", "territory_code_3", "country_full", "email_cost", "web_engaged_sub", "email_engaged_sub", "mps_flag", "opportunities", "is_sub", "mpc_flag", "downloads", "video_views", "logins", "pageviews", "guide_to_markets_interactions", "market_insights_email_signups", "registrations", "podcast_plays", "bond_ladder_tool_analysis", "portfolio_insights_tool_analysis", "investment_comparison_tool_analysis", "target_date_compass_tool_analysis", "bond_ladder_tool_report", "portfolio_insights_tool_report", "investment_comparison_tool_report", "target_date_compass_tool_report", "total_tool_report", "total_tool_analysis", "total" ] }, "symbols" : { "email_cost" : "decimal(2,1)", "subs_frequency" : "varchar(13)", "is_sub" : "double", "is_bounce" : "double", "primary_topics" : "varchar", "investment_comparison_tool_report" : "double", "registrations" : "double", "is_open" : "double", "subregion" : "varchar", "bond_ladder_tool_analysis" : "double", "web_engaged_sub" : "decimal(2,1)", "mps_flag" : "decimal(2,1)", "sfmc_activity_sub_type" : "varchar", "sfmc_activity_type" : "varchar", "sfmc_activity_id" : "varchar", "bond_ladder_tool_report" : "double", "total_tool_report" : "double", "territory_code_3" : "varchar", "secondary_topics" : "varchar", "territory_code_1" : "varchar", "territory_code_2" : "varchar", "territory_code_0" : "varchar", "total_tool_analysis" : "double", "campaign_id" : "varchar", "is_inferred_open" : "double", "salesforce_contact_id" : "varchar", "region" : "varchar", "logins" : "double", "is_delivered" : "double", "random" : "double", "pageviews" : "double", "downloads" : "double", "is_link_clicked" : "double", "mpc_flag" : "integer", "portfolio_insights_tool_report" : "double", "total_opens" : "double", "target_date_compass_tool_report" : "double", "gl_lob" : "varchar", "video_views" : "double", "sales_channel" : "varchar", "is_hard_bounce" : "double", "podcast_plays" : "double", "target_date_compass_tool_analysis" : "double", "is_opt_out" : "double", "investment_comparison_tool_analysis" : "double", "channel_name" : "varchar", "opportunities" : "bigint", "company_id_l1" : "varchar", "is_sent" : "double", "day" : "date", "market_insights_email_signups" : "double", "territory_code" : "varchar", "email_engaged_sub" : "decimal(2,1)", "email_id" : "varchar", "portfolio_insights_tool_analysis" : "double", "country_full" : "varchar", "campaign_name" : "varchar", "total" : "double", "url_alias" : "varchar", "tier" : "varchar", "url_clicked_personalized" : "varchar", "guide_to_markets_interactions" : "double" }, "partitioning" : { "connectorHandle" : { "@type" : "$remote", "partitioning" : "SINGLE", "function" : "SINGLE" } }, "partitionedSources" : [ ], "partitioningScheme" : { "partitioning" : { "handle" : { "connectorHandle" : { "@type" : "$remote", "partitioning" : "SINGLE", "function" : "SINGLE" } }, "arguments" : [ ] }, "outputLayout" : [ "salesforce_contact_id", "email_id", "campaign_id", "sfmc_activity_id", "campaign_name", "sfmc_activity_type", "sfmc_activity_sub_type", "day", "is_delivered", "is_open", "is_inferred_open", "total_opens", "is_link_clicked", "is_hard_bounce", "is_bounce", "is_opt_out", "is_sent", "gl_lob", "region", "subregion", "company_id_l1", "tier", "sales_channel", "channel_name", "primary_topics", "secondary_topics", "subs_frequency", "url_clicked_personalized", "url_alias", "territory_code", "territory_code_0", "territory_code_1", "territory_code_2", "territory_code_3", "country_full", "email_cost", "web_engaged_sub", "email_engaged_sub", "mps_flag", "opportunities", "is_sub", "mpc_flag", "downloads", "video_views", "logins", "pageviews", "guide_to_markets_interactions", "market_insights_email_signups", "registrations", "podcast_plays", "bond_ladder_tool_analysis", "portfolio_insights_tool_analysis", "investment_comparison_tool_analysis", "target_date_compass_tool_analysis", "bond_ladder_tool_report", "portfolio_insights_tool_report", "investment_comparison_tool_report", "target_date_compass_tool_report", "total_tool_report", "total_tool_analysis", "total" ], "replicateNullsAndAny" : false, "bucketToPartition" : [ 0 ] }, "stageExecutionDescriptor" : { "strategy" : "UNGROUPED_EXECUTION", "groupedExecutionScanNodes" : [ ] }, "statsAndCosts" : { "stats" : { }, "costs" : { } }, "jsonRepresentation" : "{\n \"id\" : \"7\",\n \"name\" : \"Output\",\n \"identifier\" : \"[salesforce_contact_id, email_id, campaign_id, sfmc_activity_id, campaign_name, sfmc_activity_type, sfmc_activity_sub_type, day, is_delivered, is_open, is_inferred_open, total_opens, is_link_clicked, is_hard_bounce, is_bounce, is_opt_out, is_sent, gl_lob, region, subregion, company_id_l1, tier, sales_channel, channel_name, primary_topics, secondary_topics, subs_frequency, url_clicked_personalized, url_alias_80, territory_code_1, territory_code_2, territory_code_3, territory_code_4, territory_code_5, country_full, email_cost, web_engaged_sub, email_engaged_sub, mps_flag, opportunities, is_sub, mpc_flag, downloads, video_views, logins, pageviews, guide_to_markets_interactions, market_insights_email_signups, registrations, podcast_plays, bond_ladder_tool_analysis, portfolio_insights_tool_analysis, investment_comparison_tool_analysis, target_date_compass_tool_analysis, bond_ladder_tool_report, portfolio_insights_tool_report, investment_comparison_tool_report, target_date_compass_tool_report, total_tool_report, total_tool_analysis, total]\",\n \"details\" : \"url_alias_80 := url_alias\nterritory_code_1 := territory_code\nterritory_code_2 := territory_code_0\nterritory_code_3 := territory_code_1\nterritory_code_4 := territory_code_2\nterritory_code_5 := territory_code_3\n\",\n \"children\" : [ {\n \"id\" : \"4\",\n \"name\" : \"Project\",\n \"identifier\" : \"[]\",\n \"details\" : \"\",\n \"children\" : [ {\n \"id\" : \"166\",\n \"name\" : \"RemoteMerge\",\n \"identifier\" : \"[1]\",\n \"details\" : \"\",\n \"children\" : [ ],\n \"remoteSources\" : [ \"1\" ]\n } ],\n \"remoteSources\" : [ ]\n } ],\n \"remoteSources\" : [ ]\n}" }, "types" : [ "varchar", "varchar", "varchar", "varchar", "varchar", "varchar", "varchar", "date", "double", "double", "double", "double", "double", "double", "double", "double", "double", "varchar", "varchar", "varchar", "varchar", "varchar", "varchar", "varchar", "varchar", "varchar", "varchar(13)", "varchar", "varchar", "varchar", "varchar", "varchar", "varchar", "varchar", "varchar", "decimal(2,1)", "decimal(2,1)", "decimal(2,1)", "decimal(2,1)", "bigint", "double", "integer", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double" ], "stageStats" : { "schedulingComplete" : "2020-03-16T14:31:53.202Z", "getSplitDistribution" : { "count" : 0.0, "total" : 0.0, "p01" : "NaN", "p05" : "NaN", "p10" : "NaN", "p25" : "NaN", "p50" : "NaN", "p75" : "NaN", "p90" : "NaN", "p95" : "NaN", "p99" : "NaN", "min" : "NaN", "max" : "NaN", "avg" : "NaN" }, "totalTasks" : 1, "runningTasks" : 0, "completedTasks" : 1, "totalDrivers" : 1, "queuedDrivers" : 0, "runningDrivers" : 0, "blockedDrivers" : 1, "completedDrivers" : 0, "cumulativeUserMemory" : 0.0, "userMemoryReservation" : "0B", "revocableMemoryReservation" : "0B", "totalMemoryReservation" : "0B", "peakUserMemoryReservation" : "0B", "peakRevocableMemoryReservation" : "0B", "totalScheduledTime" : "235.17us", "totalCpuTime" : "230.00us", "totalBlockedTime" : "30.98s", "fullyBlocked" : false, "blockedReasons" : [ ], "physicalInputDataSize" : "0B", "physicalInputPositions" : 0, "physicalInputReadTime" : "0.00ns", "internalNetworkInputDataSize" : "0B", "internalNetworkInputPositions" : 0, "rawInputDataSize" : "0B", "rawInputPositions" : 0, "processedInputDataSize" : "0B", "processedInputPositions" : 0, "bufferedDataSize" : "0B", "outputDataSize" : "0B", "outputPositions" : 0, "physicalWrittenDataSize" : "0B", "gcInfo" : { "stageId" : 0, "tasks" : 1, "fullGcTasks" : 0, "minFullGcSec" : 0, "maxFullGcSec" : 0, "totalFullGcSec" : 0, "averageFullGcSec" : 0 }, "operatorSummaries" : [ { "stageId" : 0, "pipelineId" : 0, "operatorId" : 0, "planNodeId" : "166", "operatorType" : "MergeOperator", "totalDrivers" : 1, "addInputCalls" : 0, "addInputWall" : "0.00ns", "addInputCpu" : "0.00ns", "physicalInputDataSize" : "0B", "physicalInputPositions" : 0, "internalNetworkInputDataSize" : "0B", "internalNetworkInputPositions" : 0, "rawInputDataSize" : "0B", "inputDataSize" : "0B", "inputPositions" : 0, "sumSquaredInputPositions" : 0.0, "getOutputCalls" : 1, "getOutputWall" : "122.66us", "getOutputCpu" : "115.00us", "outputDataSize" : "0B", "outputPositions" : 0, "physicalWrittenDataSize" : "0B", "blockedWall" : "0.00ns", "finishCalls" : 0, "finishWall" : "0.00ns", "finishCpu" : "0.00ns", "userMemoryReservation" : "0B", "revocableMemoryReservation" : "0B", "systemMemoryReservation" : "0B", "peakUserMemoryReservation" : "0B", "peakSystemMemoryReservation" : "0B", "peakRevocableMemoryReservation" : "0B", "peakTotalMemoryReservation" : "0B", "spilledDataSize" : "0B" }, { "stageId" : 0, "pipelineId" : 0, "operatorId" : 1, "planNodeId" : "4", "operatorType" : "FilterAndProjectOperator", "totalDrivers" : 1, "addInputCalls" : 0, "addInputWall" : "0.00ns", "addInputCpu" : "0.00ns", "physicalInputDataSize" : "0B", "physicalInputPositions" : 0, "internalNetworkInputDataSize" : "0B", "internalNetworkInputPositions" : 0, "rawInputDataSize" : "0B", "inputDataSize" : "0B", "inputPositions" : 0, "sumSquaredInputPositions" : 0.0, "getOutputCalls" : 2, "getOutputWall" : "80.29us", "getOutputCpu" : "83.00us", "outputDataSize" : "0B", "outputPositions" : 0, "physicalWrittenDataSize" : "0B", "blockedWall" : "0.00ns", "finishCalls" : 0, "finishWall" : "0.00ns", "finishCpu" : "0.00ns", "userMemoryReservation" : "0B", "revocableMemoryReservation" : "0B", "systemMemoryReservation" : "0B", "peakUserMemoryReservation" : "0B", "peakSystemMemoryReservation" : "0B", "peakRevocableMemoryReservation" : "0B", "peakTotalMemoryReservation" : "0B", "spilledDataSize" : "0B" }, { "stageId" : 0, "pipelineId" : 0, "operatorId" : 2, "planNodeId" : "7", "operatorType" : "TaskOutputOperator", "totalDrivers" : 1, "addInputCalls" : 0, "addInputWall" : "0.00ns", "addInputCpu" : "0.00ns", "physicalInputDataSize" : "0B", "physicalInputPositions" : 0, "internalNetworkInputDataSize" : "0B", "internalNetworkInputPositions" : 0, "rawInputDataSize" : "0B", "inputDataSize" : "0B", "inputPositions" : 0, "sumSquaredInputPositions" : 0.0, "getOutputCalls" : 0, "getOutputWall" : "0.00ns", "getOutputCpu" : "0.00ns", "outputDataSize" : "0B", "outputPositions" : 0, "physicalWrittenDataSize" : "0B", "blockedWall" : "0.00ns", "finishCalls" : 0, "finishWall" : "0.00ns", "finishCpu" : "0.00ns", "userMemoryReservation" : "0B", "revocableMemoryReservation" : "0B", "systemMemoryReservation" : "0B", "peakUserMemoryReservation" : "0B", "peakSystemMemoryReservation" : "0B", "peakRevocableMemoryReservation" : "0B", "peakTotalMemoryReservation" : "0B", "spilledDataSize" : "0B" } ] }, "tasks" : [ { "taskStatus" : { "taskId" : "20200316_143152_00013_fbfw4.0.0", "taskInstanceId" : "0341d2ea-a049-4ff8-91fc-dc5cdbef29b7", "version" : 337, "state" : "ABORTED", "self" : "http://localhost:8080/v1/task/20200316_143152_00013_fbfw4.0.0", "nodeId" : "ffffffff-ffff-ffff-ffff-ffffffffffff", "completedDriverGroups" : [ ], "failures" : [ ], "queuedPartitionedDrivers" : 0, "runningPartitionedDrivers" : 0, "outputBufferOverutilized" : false, "physicalWrittenDataSize" : "0B", "memoryReservation" : "0B", "systemMemoryReservation" : "0B", "revocableMemoryReservation" : "0B", "fullGcCount" : 0, "fullGcTime" : "0.00ns" }, "lastHeartbeat" : "2020-03-16T14:32:23.705Z", "outputBuffers" : { "type" : "PARTITIONED", "state" : "NO_MORE_BUFFERS", "canAddBuffers" : false, "canAddPages" : true, "totalBufferedBytes" : 0, "totalBufferedPages" : 0, "totalRowsSent" : 0, "totalPagesSent" : 0, "buffers" : [ ] }, "noMoreSplits" : [ "166" ], "stats" : { "createTime" : "2020-03-16T14:31:53.216Z", "firstStartTime" : "2020-03-16T14:31:53.277Z", "lastStartTime" : "2020-03-16T14:31:53.277Z", "lastEndTime" : "2020-03-16T14:32:24.256Z", "endTime" : "2020-03-16T14:32:24.256Z", "elapsedTime" : "31.00s", "queuedTime" : "26.69ms", "totalDrivers" : 1, "queuedDrivers" : 0, "queuedPartitionedDrivers" : 0, "runningDrivers" : 0, "runningPartitionedDrivers" : 0, "blockedDrivers" : 1, "completedDrivers" : 0, "cumulativeUserMemory" : 0.0, "userMemoryReservation" : "0B", "revocableMemoryReservation" : "0B", "systemMemoryReservation" : "0B", "totalScheduledTime" : "235.17us", "totalCpuTime" : "230.00us", "totalBlockedTime" : "30.98s", "fullyBlocked" : false, "blockedReasons" : [ ], "physicalInputDataSize" : "0B", "physicalInputPositions" : 0, "physicalInputReadTime" : "0.00ns", "internalNetworkInputDataSize" : "0B", "internalNetworkInputPositions" : 0, "rawInputDataSize" : "0B", "rawInputPositions" : 0, "processedInputDataSize" : "0B", "processedInputPositions" : 0, "outputDataSize" : "0B", "outputPositions" : 0, "physicalWrittenDataSize" : "0B", "fullGcCount" : 0, "fullGcTime" : "0.00ns", "pipelines" : [ { "pipelineId" : 0, "firstStartTime" : "2020-03-16T14:31:53.277Z", "lastStartTime" : "2020-03-16T14:31:53.277Z", "lastEndTime" : "2020-03-16T14:32:24.256Z", "inputPipeline" : true, "outputPipeline" : true, "totalDrivers" : 1, "queuedDrivers" : 0, "queuedPartitionedDrivers" : 0, "runningDrivers" : 0, "runningPartitionedDrivers" : 0, "blockedDrivers" : 1, "completedDrivers" : 0, "userMemoryReservation" : "0B", "revocableMemoryReservation" : "0B", "systemMemoryReservation" : "0B", "queuedTime" : { "count" : 1.0, "total" : 1.3240519E7, "p01" : 1.3240519E7, "p05" : 1.3240519E7, "p10" : 1.3240519E7, "p25" : 1.3240519E7, "p50" : 1.3240519E7, "p75" : 1.3240519E7, "p90" : 1.3240519E7, "p95" : 1.3240519E7, "p99" : 1.3240519E7, "min" : 1.3240519E7, "max" : 1.3240519E7, "avg" : 1.3240519E7 }, "elapsedTime" : { "count" : 1.0, "total" : 3.0991389203E10, "p01" : 3.0991389203E10, "p05" : 3.0991389203E10, "p10" : 3.0991389203E10, "p25" : 3.0991389203E10, "p50" : 3.0991389203E10, "p75" : 3.0991389203E10, "p90" : 3.0991389203E10, "p95" : 3.0991389203E10, "p99" : 3.0991389203E10, "min" : 3.0991389203E10, "max" : 3.0991389203E10, "avg" : 3.0991389203E10 }, "totalScheduledTime" : "235.17us", "totalCpuTime" : "230.00us", "totalBlockedTime" : "30.98s", "fullyBlocked" : true, "blockedReasons" : [ ], "physicalInputDataSize" : "0B", "physicalInputPositions" : 0, "physicalInputReadTime" : "0.00ns", "internalNetworkInputDataSize" : "0B", "internalNetworkInputPositions" : 0, "rawInputDataSize" : "0B", "rawInputPositions" : 0, "processedInputDataSize" : "0B", "processedInputPositions" : 0, "outputDataSize" : "0B", "outputPositions" : 0, "physicalWrittenDataSize" : "0B", "operatorSummaries" : [ { "stageId" : 0, "pipelineId" : 0, "operatorId" : 0, "planNodeId" : "166", "operatorType" : "MergeOperator", "totalDrivers" : 1, "addInputCalls" : 0, "addInputWall" : "0.00ns", "addInputCpu" : "0.00ns", "physicalInputDataSize" : "0B", "physicalInputPositions" : 0, "internalNetworkInputDataSize" : "0B", "internalNetworkInputPositions" : 0, "rawInputDataSize" : "0B", "inputDataSize" : "0B", "inputPositions" : 0, "sumSquaredInputPositions" : 0.0, "getOutputCalls" : 1, "getOutputWall" : "122.66us", "getOutputCpu" : "115.00us", "outputDataSize" : "0B", "outputPositions" : 0, "physicalWrittenDataSize" : "0B", "blockedWall" : "0.00ns", "finishCalls" : 0, "finishWall" : "0.00ns", "finishCpu" : "0.00ns", "userMemoryReservation" : "0B", "revocableMemoryReservation" : "0B", "systemMemoryReservation" : "0B", "peakUserMemoryReservation" : "0B", "peakSystemMemoryReservation" : "0B", "peakRevocableMemoryReservation" : "0B", "peakTotalMemoryReservation" : "0B", "spilledDataSize" : "0B" }, { "stageId" : 0, "pipelineId" : 0, "operatorId" : 1, "planNodeId" : "4", "operatorType" : "FilterAndProjectOperator", "totalDrivers" : 1, "addInputCalls" : 0, "addInputWall" : "0.00ns", "addInputCpu" : "0.00ns", "physicalInputDataSize" : "0B", "physicalInputPositions" : 0, "internalNetworkInputDataSize" : "0B", "internalNetworkInputPositions" : 0, "rawInputDataSize" : "0B", "inputDataSize" : "0B", "inputPositions" : 0, "sumSquaredInputPositions" : 0.0, "getOutputCalls" : 2, "getOutputWall" : "80.29us", "getOutputCpu" : "83.00us", "outputDataSize" : "0B", "outputPositions" : 0, "physicalWrittenDataSize" : "0B", "blockedWall" : "0.00ns", "finishCalls" : 0, "finishWall" : "0.00ns", "finishCpu" : "0.00ns", "userMemoryReservation" : "0B", "revocableMemoryReservation" : "0B", "systemMemoryReservation" : "0B", "peakUserMemoryReservation" : "0B", "peakSystemMemoryReservation" : "0B", "peakRevocableMemoryReservation" : "0B", "peakTotalMemoryReservation" : "0B", "spilledDataSize" : "0B" }, { "stageId" : 0, "pipelineId" : 0, "operatorId" : 2, "planNodeId" : "7", "operatorType" : "TaskOutputOperator", "totalDrivers" : 1, "addInputCalls" : 0, "addInputWall" : "0.00ns", "addInputCpu" : "0.00ns", "physicalInputDataSize" : "0B", "physicalInputPositions" : 0, "internalNetworkInputDataSize" : "0B", "internalNetworkInputPositions" : 0, "rawInputDataSize" : "0B", "inputDataSize" : "0B", "inputPositions" : 0, "sumSquaredInputPositions" : 0.0, "getOutputCalls" : 0, "getOutputWall" : "0.00ns", "getOutputCpu" : "0.00ns", "outputDataSize" : "0B", "outputPositions" : 0, "physicalWrittenDataSize" : "0B", "blockedWall" : "0.00ns", "finishCalls" : 0, "finishWall" : "0.00ns", "finishCpu" : "0.00ns", "userMemoryReservation" : "0B", "revocableMemoryReservation" : "0B", "systemMemoryReservation" : "0B", "peakUserMemoryReservation" : "0B", "peakSystemMemoryReservation" : "0B", "peakRevocableMemoryReservation" : "0B", "peakTotalMemoryReservation" : "0B", "spilledDataSize" : "0B" } ], "drivers" : [ ] } ] }, "needsPlan" : false } ], "subStages" : [ { "stageId" : "20200316_143152_00013_fbfw4.1", "state" : "ABORTED", "plan" : { "id" : "1", "root" : { "@type" : "exchange", "id" : "204", "type" : "GATHER", "scope" : "LOCAL", "partitioningScheme" : { "partitioning" : { "handle" : { "connectorHandle" : { "@type" : "$remote", "partitioning" : "FIXED", "function" : "UNKNOWN" } }, "arguments" : [ ] }, "outputLayout" : [ "random", "salesforce_contact_id", "email_id", "campaign_id", "sfmc_activity_id", "campaign_name", "sfmc_activity_type", "sfmc_activity_sub_type", "day", "is_delivered", "is_open", "is_inferred_open", "total_opens", "is_link_clicked", "is_hard_bounce", "is_bounce", "is_opt_out", "is_sent", "gl_lob", "region", "subregion", "company_id_l1", "tier", "sales_channel", "channel_name", "primary_topics", "secondary_topics", "subs_frequency", "url_clicked_personalized", "url_alias", "territory_code", "territory_code_0", "territory_code_1", "territory_code_2", "territory_code_3", "country_full", "email_cost", "web_engaged_sub", "email_engaged_sub", "mps_flag", "opportunities", "is_sub", "mpc_flag", "downloads", "video_views", "logins", "pageviews", "guide_to_markets_interactions", "market_insights_email_signups", "registrations", "podcast_plays", "bond_ladder_tool_analysis", "portfolio_insights_tool_analysis", "investment_comparison_tool_analysis", "target_date_compass_tool_analysis", "bond_ladder_tool_report", "portfolio_insights_tool_report", "investment_comparison_tool_report", "target_date_compass_tool_report", "total_tool_report", "total_tool_analysis", "total" ], "replicateNullsAndAny" : false }, "sources" : [ { "@type" : "sort", "id" : "167", "source" : { "@type" : "remoteSource", "id" : "165", "sourceFragmentIds" : [ "2" ], "outputs" : [ "random", "salesforce_contact_id", "email_id", "campaign_id", "sfmc_activity_id", "campaign_name", "sfmc_activity_type", "sfmc_activity_sub_type", "day", "is_delivered", "is_open", "is_inferred_open", "total_opens", "is_link_clicked", "is_hard_bounce", "is_bounce", "is_opt_out", "is_sent", "gl_lob", "region", "subregion", "company_id_l1", "tier", "sales_channel", "channel_name", "primary_topics", "secondary_topics", "subs_frequency", "url_clicked_personalized", "url_alias", "territory_code", "territory_code_0", "territory_code_1", "territory_code_2", "territory_code_3", "country_full", "email_cost", "web_engaged_sub", "email_engaged_sub", "mps_flag", "opportunities", "is_sub", "mpc_flag", "downloads", "video_views", "logins", "pageviews", "guide_to_markets_interactions", "market_insights_email_signups", "registrations", "podcast_plays", "bond_ladder_tool_analysis", "portfolio_insights_tool_analysis", "investment_comparison_tool_analysis", "target_date_compass_tool_analysis", "bond_ladder_tool_report", "portfolio_insights_tool_report", "investment_comparison_tool_report", "target_date_compass_tool_report", "total_tool_report", "total_tool_analysis", "total" ], "exchangeType" : "REPARTITION" }, "orderingScheme" : { "orderBy" : [ "random" ], "orderings" : { "random" : "ASC_NULLS_LAST" } }, "partial" : true } ], "inputs" : [ [ "random", "salesforce_contact_id", "email_id", "campaign_id", "sfmc_activity_id", "campaign_name", "sfmc_activity_type", "sfmc_activity_sub_type", "day", "is_delivered", "is_open", "is_inferred_open", "total_opens", "is_link_clicked", "is_hard_bounce", "is_bounce", "is_opt_out", "is_sent", "gl_lob", "region", "subregion", "company_id_l1", "tier", "sales_channel", "channel_name", "primary_topics", "secondary_topics", "subs_frequency", "url_clicked_personalized", "url_alias", "territory_code", "territory_code_0", "territory_code_1", "territory_code_2", "territory_code_3", "country_full", "email_cost", "web_engaged_sub", "email_engaged_sub", "mps_flag", "opportunities", "is_sub", "mpc_flag", "downloads", "video_views", "logins", "pageviews", "guide_to_markets_interactions", "market_insights_email_signups", "registrations", "podcast_plays", "bond_ladder_tool_analysis", "portfolio_insights_tool_analysis", "investment_comparison_tool_analysis", "target_date_compass_tool_analysis", "bond_ladder_tool_report", "portfolio_insights_tool_report", "investment_comparison_tool_report", "target_date_compass_tool_report", "total_tool_report", "total_tool_analysis", "total" ] ], "orderingScheme" : { "orderBy" : [ "random" ], "orderings" : { "random" : "ASC_NULLS_LAST" } } }, "symbols" : { "email_cost" : "decimal(2,1)", "subs_frequency" : "varchar(13)", "is_sub" : "double", "is_bounce" : "double", "primary_topics" : "varchar", "investment_comparison_tool_report" : "double", "registrations" : "double", "is_open" : "double", "subregion" : "varchar", "bond_ladder_tool_analysis" : "double", "web_engaged_sub" : "decimal(2,1)", "mps_flag" : "decimal(2,1)", "sfmc_activity_sub_type" : "varchar", "sfmc_activity_type" : "varchar", "sfmc_activity_id" : "varchar", "bond_ladder_tool_report" : "double", "total_tool_report" : "double", "territory_code_3" : "varchar", "secondary_topics" : "varchar", "territory_code_1" : "varchar", "territory_code_2" : "varchar", "territory_code_0" : "varchar", "total_tool_analysis" : "double", "campaign_id" : "varchar", "is_inferred_open" : "double", "salesforce_contact_id" : "varchar", "region" : "varchar", "logins" : "double", "is_delivered" : "double", "random" : "double", "pageviews" : "double", "downloads" : "double", "is_link_clicked" : "double", "mpc_flag" : "integer", "portfolio_insights_tool_report" : "double", "total_opens" : "double", "target_date_compass_tool_report" : "double", "gl_lob" : "varchar", "video_views" : "double", "sales_channel" : "varchar", "is_hard_bounce" : "double", "podcast_plays" : "double", "target_date_compass_tool_analysis" : "double", "is_opt_out" : "double", "investment_comparison_tool_analysis" : "double", "channel_name" : "varchar", "opportunities" : "bigint", "company_id_l1" : "varchar", "is_sent" : "double", "day" : "date", "market_insights_email_signups" : "double", "territory_code" : "varchar", "email_engaged_sub" : "decimal(2,1)", "email_id" : "varchar", "portfolio_insights_tool_analysis" : "double", "country_full" : "varchar", "campaign_name" : "varchar", "total" : "double", "url_alias" : "varchar", "tier" : "varchar", "url_clicked_personalized" : "varchar", "guide_to_markets_interactions" : "double" }, "partitioning" : { "connectorHandle" : { "@type" : "$remote", "partitioning" : "FIXED", "function" : "ROUND_ROBIN" } }, "partitionedSources" : [ ], "partitioningScheme" : { "partitioning" : { "handle" : { "connectorHandle" : { "@type" : "$remote", "partitioning" : "SINGLE", "function" : "SINGLE" } }, "arguments" : [ ] }, "outputLayout" : [ "random", "salesforce_contact_id", "email_id", "campaign_id", "sfmc_activity_id", "campaign_name", "sfmc_activity_type", "sfmc_activity_sub_type", "day", "is_delivered", "is_open", "is_inferred_open", "total_opens", "is_link_clicked", "is_hard_bounce", "is_bounce", "is_opt_out", "is_sent", "gl_lob", "region", "subregion", "company_id_l1", "tier", "sales_channel", "channel_name", "primary_topics", "secondary_topics", "subs_frequency", "url_clicked_personalized", "url_alias", "territory_code", "territory_code_0", "territory_code_1", "territory_code_2", "territory_code_3", "country_full", "email_cost", "web_engaged_sub", "email_engaged_sub", "mps_flag", "opportunities", "is_sub", "mpc_flag", "downloads", "video_views", "logins", "pageviews", "guide_to_markets_interactions", "market_insights_email_signups", "registrations", "podcast_plays", "bond_ladder_tool_analysis", "portfolio_insights_tool_analysis", "investment_comparison_tool_analysis", "target_date_compass_tool_analysis", "bond_ladder_tool_report", "portfolio_insights_tool_report", "investment_comparison_tool_report", "target_date_compass_tool_report", "total_tool_report", "total_tool_analysis", "total" ], "replicateNullsAndAny" : false, "bucketToPartition" : [ 0 ] }, "stageExecutionDescriptor" : { "strategy" : "UNGROUPED_EXECUTION", "groupedExecutionScanNodes" : [ ] }, "statsAndCosts" : { "stats" : { }, "costs" : { } }, "jsonRepresentation" : "{\n \"id\" : \"204\",\n \"name\" : \"LocalMerge\",\n \"identifier\" : \"[random ASC_NULLS_LAST]\",\n \"details\" : \"\",\n \"children\" : [ {\n \"id\" : \"167\",\n \"name\" : \"PartialSort\",\n \"identifier\" : \"[random ASC_NULLS_LAST]\",\n \"details\" : \"\",\n \"children\" : [ {\n \"id\" : \"165\",\n \"name\" : \"RemoteSource\",\n \"identifier\" : \"[2]\",\n \"details\" : \"\",\n \"children\" : [ ],\n \"remoteSources\" : [ \"2\" ]\n } ],\n \"remoteSources\" : [ ]\n } ],\n \"remoteSources\" : [ ]\n}" }, "types" : [ "double", "varchar", "varchar", "varchar", "varchar", "varchar", "varchar", "varchar", "date", "double", "double", "double", "double", "double", "double", "double", "double", "double", "varchar", "varchar", "varchar", "varchar", "varchar", "varchar", "varchar", "varchar", "varchar", "varchar(13)", "varchar", "varchar", "varchar", "varchar", "varchar", "varchar", "varchar", "varchar", "decimal(2,1)", "decimal(2,1)", "decimal(2,1)", "decimal(2,1)", "bigint", "double", "integer", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double" ], "stageStats" : { "schedulingComplete" : "2020-03-16T14:31:53.191Z", "getSplitDistribution" : { "count" : 0.0, "total" : 0.0, "p01" : "NaN", "p05" : "NaN", "p10" : "NaN", "p25" : "NaN", "p50" : "NaN", "p75" : "NaN", "p90" : "NaN", "p95" : "NaN", "p99" : "NaN", "min" : "NaN", "max" : "NaN", "avg" : "NaN" }, "totalTasks" : 1, "runningTasks" : 0, "completedTasks" : 1, "totalDrivers" : 17, "queuedDrivers" : 0, "runningDrivers" : 0, "blockedDrivers" : 17, "completedDrivers" : 0, "cumulativeUserMemory" : 7.46334276797507E14, "userMemoryReservation" : "53678376821B", "revocableMemoryReservation" : "0B", "totalMemoryReservation" : "53678376821B", "peakUserMemoryReservation" : "52565356430B", "peakRevocableMemoryReservation" : "0B", "totalScheduledTime" : "58.50s", "totalCpuTime" : "16.52s", "totalBlockedTime" : "7.75m", "fullyBlocked" : false, "blockedReasons" : [ ], "physicalInputDataSize" : "0B", "physicalInputPositions" : 0, "physicalInputReadTime" : "0.00ns", "internalNetworkInputDataSize" : "2110724193B", "internalNetworkInputPositions" : 5197502, "rawInputDataSize" : "2110724193B", "rawInputPositions" : 5197502, "processedInputDataSize" : "2374995020B", "processedInputPositions" : 5197502, "bufferedDataSize" : "0B", "outputDataSize" : "0B", "outputPositions" : 0, "physicalWrittenDataSize" : "0B", "gcInfo" : { "stageId" : 1, "tasks" : 1, "fullGcTasks" : 0, "minFullGcSec" : 0, "maxFullGcSec" : 0, "totalFullGcSec" : 0, "averageFullGcSec" : 0 }, "operatorSummaries" : [ { "stageId" : 1, "pipelineId" : 0, "operatorId" : 0, "planNodeId" : "165", "operatorType" : "ExchangeOperator", "totalDrivers" : 16, "addInputCalls" : 0, "addInputWall" : "0.00ns", "addInputCpu" : "0.00ns", "physicalInputDataSize" : "0B", "physicalInputPositions" : 0, "internalNetworkInputDataSize" : "2110724193B", "internalNetworkInputPositions" : 5197502, "rawInputDataSize" : "2110724193B", "inputDataSize" : "2374995020B", "inputPositions" : 5197502, "sumSquaredInputPositions" : 1.695206720428E12, "getOutputCalls" : 5308, "getOutputWall" : "42.85s", "getOutputCpu" : "11.51s", "outputDataSize" : "2374995020B", "outputPositions" : 5197502, "physicalWrittenDataSize" : "0B", "blockedWall" : "7.22m", "finishCalls" : 0, "finishWall" : "0.00ns", "finishCpu" : "0.00ns", "userMemoryReservation" : "0B", "revocableMemoryReservation" : "0B", "systemMemoryReservation" : "0B", "peakUserMemoryReservation" : "0B", "peakSystemMemoryReservation" : "0B", "peakRevocableMemoryReservation" : "0B", "peakTotalMemoryReservation" : "0B", "spilledDataSize" : "0B", "info" : { "@type" : "exchangeClientStatus", "bufferedBytes" : 0, "maxBufferedBytes" : 12581962, "averageBytesPerRequest" : 5704461, "successfulRequestsCount" : 5920, "bufferedPages" : 0, "noMoreLocations" : true, "pageBufferClientStatuses" : [ ] } }, { "stageId" : 1, "pipelineId" : 1, "operatorId" : 0, "planNodeId" : "204", "operatorType" : "LocalMergeSourceOperator", "totalDrivers" : 1, "addInputCalls" : 0, "addInputWall" : "0.00ns", "addInputCpu" : "0.00ns", "physicalInputDataSize" : "0B", "physicalInputPositions" : 0, "internalNetworkInputDataSize" : "0B", "internalNetworkInputPositions" : 0, "rawInputDataSize" : "0B", "inputDataSize" : "0B", "inputPositions" : 0, "sumSquaredInputPositions" : 0.0, "getOutputCalls" : 1, "getOutputWall" : "1.67ms", "getOutputCpu" : "234.00us", "outputDataSize" : "0B", "outputPositions" : 0, "physicalWrittenDataSize" : "0B", "blockedWall" : "0.00ns", "finishCalls" : 0, "finishWall" : "0.00ns", "finishCpu" : "0.00ns", "userMemoryReservation" : "0B", "revocableMemoryReservation" : "0B", "systemMemoryReservation" : "0B", "peakUserMemoryReservation" : "0B", "peakSystemMemoryReservation" : "0B", "peakRevocableMemoryReservation" : "0B", "peakTotalMemoryReservation" : "0B", "spilledDataSize" : "0B" }, { "stageId" : 1, "pipelineId" : 0, "operatorId" : 1, "planNodeId" : "167", "operatorType" : "OrderByOperator", "totalDrivers" : 16, "addInputCalls" : 5214, "addInputWall" : "13.95s", "addInputCpu" : "4.53s", "physicalInputDataSize" : "0B", "physicalInputPositions" : 0, "internalNetworkInputDataSize" : "0B", "internalNetworkInputPositions" : 0, "rawInputDataSize" : "0B", "inputDataSize" : "2374995020B", "inputPositions" : 5197502, "sumSquaredInputPositions" : 1.695206720428E12, "getOutputCalls" : 10924, "getOutputWall" : "1.42s", "getOutputCpu" : "297.01ms", "outputDataSize" : "0B", "outputPositions" : 0, "physicalWrittenDataSize" : "0B", "blockedWall" : "0.00ns", "finishCalls" : 0, "finishWall" : "0.00ns", "finishCpu" : "0.00ns", "userMemoryReservation" : "53678376821B", "revocableMemoryReservation" : "0B", "systemMemoryReservation" : "0B", "peakUserMemoryReservation" : "3783289414B", "peakSystemMemoryReservation" : "0B", "peakRevocableMemoryReservation" : "0B", "peakTotalMemoryReservation" : "3783289414B", "spilledDataSize" : "0B" }, { "stageId" : 1, "pipelineId" : 1, "operatorId" : 1, "planNodeId" : "204", "operatorType" : "TaskOutputOperator", "totalDrivers" : 1, "addInputCalls" : 0, "addInputWall" : "0.00ns", "addInputCpu" : "0.00ns", "physicalInputDataSize" : "0B", "physicalInputPositions" : 0, "internalNetworkInputDataSize" : "0B", "internalNetworkInputPositions" : 0, "rawInputDataSize" : "0B", "inputDataSize" : "0B", "inputPositions" : 0, "sumSquaredInputPositions" : 0.0, "getOutputCalls" : 0, "getOutputWall" : "0.00ns", "getOutputCpu" : "0.00ns", "outputDataSize" : "0B", "outputPositions" : 0, "physicalWrittenDataSize" : "0B", "blockedWall" : "0.00ns", "finishCalls" : 0, "finishWall" : "0.00ns", "finishCpu" : "0.00ns", "userMemoryReservation" : "0B", "revocableMemoryReservation" : "0B", "systemMemoryReservation" : "0B", "peakUserMemoryReservation" : "0B", "peakSystemMemoryReservation" : "0B", "peakRevocableMemoryReservation" : "0B", "peakTotalMemoryReservation" : "0B", "spilledDataSize" : "0B" }, { "stageId" : 1, "pipelineId" : 0, "operatorId" : 2, "planNodeId" : "204", "operatorType" : "LocalExchangeSinkOperator", "totalDrivers" : 16, "addInputCalls" : 0, "addInputWall" : "0.00ns", "addInputCpu" : "0.00ns", "physicalInputDataSize" : "0B", "physicalInputPositions" : 0, "internalNetworkInputDataSize" : "0B", "internalNetworkInputPositions" : 0, "rawInputDataSize" : "0B", "inputDataSize" : "0B", "inputPositions" : 0, "sumSquaredInputPositions" : 0.0, "getOutputCalls" : 0, "getOutputWall" : "0.00ns", "getOutputCpu" : "0.00ns", "outputDataSize" : "0B", "outputPositions" : 0, "physicalWrittenDataSize" : "0B", "blockedWall" : "0.00ns", "finishCalls" : 0, "finishWall" : "0.00ns", "finishCpu" : "0.00ns", "userMemoryReservation" : "0B", "revocableMemoryReservation" : "0B", "systemMemoryReservation" : "0B", "peakUserMemoryReservation" : "0B", "peakSystemMemoryReservation" : "0B", "peakRevocableMemoryReservation" : "0B", "peakTotalMemoryReservation" : "0B", "spilledDataSize" : "0B" } ] }, "tasks" : [ { "taskStatus" : { "taskId" : "20200316_143152_00013_fbfw4.1.0", "taskInstanceId" : "3d98367a-dc97-4e60-88fb-5c7aacd7ea3f", "version" : 339, "state" : "ABORTED", "self" : "http://localhost:8080/v1/task/20200316_143152_00013_fbfw4.1.0", "nodeId" : "ffffffff-ffff-ffff-ffff-ffffffffffff", "completedDriverGroups" : [ ], "failures" : [ ], "queuedPartitionedDrivers" : 0, "runningPartitionedDrivers" : 0, "outputBufferOverutilized" : false, "physicalWrittenDataSize" : "0B", "memoryReservation" : "53678376821B", "systemMemoryReservation" : "0B", "revocableMemoryReservation" : "0B", "fullGcCount" : 0, "fullGcTime" : "0.00ns" }, "lastHeartbeat" : "2020-03-16T14:32:23.677Z", "outputBuffers" : { "type" : "PARTITIONED", "state" : "NO_MORE_BUFFERS", "canAddBuffers" : false, "canAddPages" : true, "totalBufferedBytes" : 0, "totalBufferedPages" : 0, "totalRowsSent" : 0, "totalPagesSent" : 0, "buffers" : [ ] }, "noMoreSplits" : [ "165" ], "stats" : { "createTime" : "2020-03-16T14:31:53.199Z", "firstStartTime" : "2020-03-16T14:31:53.241Z", "lastStartTime" : "2020-03-16T14:31:53.249Z", "lastEndTime" : "2020-03-16T14:32:24.259Z", "endTime" : "2020-03-16T14:32:24.257Z", "elapsedTime" : "31.02s", "queuedTime" : "4.12ms", "totalDrivers" : 17, "queuedDrivers" : 0, "queuedPartitionedDrivers" : 0, "runningDrivers" : 0, "runningPartitionedDrivers" : 0, "blockedDrivers" : 17, "completedDrivers" : 0, "cumulativeUserMemory" : 7.46334276797507E14, "userMemoryReservation" : "53678376821B", "revocableMemoryReservation" : "0B", "systemMemoryReservation" : "0B", "totalScheduledTime" : "58.50s", "totalCpuTime" : "16.52s", "totalBlockedTime" : "7.75m", "fullyBlocked" : false, "blockedReasons" : [ ], "physicalInputDataSize" : "0B", "physicalInputPositions" : 0, "physicalInputReadTime" : "0.00ns", "internalNetworkInputDataSize" : "2110724193B", "internalNetworkInputPositions" : 5197502, "rawInputDataSize" : "2110724193B", "rawInputPositions" : 5197502, "processedInputDataSize" : "2374995020B", "processedInputPositions" : 5197502, "outputDataSize" : "0B", "outputPositions" : 0, "physicalWrittenDataSize" : "0B", "fullGcCount" : 0, "fullGcTime" : "0.00ns", "pipelines" : [ { "pipelineId" : 0, "firstStartTime" : "2020-03-16T14:31:53.241Z", "lastStartTime" : "2020-03-16T14:31:53.246Z", "lastEndTime" : "2020-03-16T14:32:24.257Z", "inputPipeline" : true, "outputPipeline" : false, "totalDrivers" : 16, "queuedDrivers" : 0, "queuedPartitionedDrivers" : 0, "runningDrivers" : 0, "runningPartitionedDrivers" : 0, "blockedDrivers" : 16, "completedDrivers" : 0, "userMemoryReservation" : "53678376821B", "revocableMemoryReservation" : "0B", "systemMemoryReservation" : "0B", "queuedTime" : { "count" : 16.0, "total" : 8.5914814E7, "p01" : 1993576.0, "p05" : 1993576.0, "p10" : 3077322.0, "p25" : 4528397.0, "p50" : 5644927.0, "p75" : 6805577.0, "p90" : 7248581.0, "p95" : 7502447.0, "p99" : 7502447.0, "min" : 1993576.0, "max" : 7502447.0, "avg" : 5369675.875 }, "elapsedTime" : { "count" : 16.0, "total" : 4.96304493551E11, "p01" : 3.1018420264E10, "p05" : 3.1018420264E10, "p10" : 3.1018498729E10, "p25" : 3.1018708114E10, "p50" : 3.101909682E10, "p75" : 3.1019422478E10, "p90" : 3.1019498704E10, "p95" : 3.1019541759E10, "p99" : 3.1019541759E10, "min" : 3.1018420264E10, "max" : 3.1019541759E10, "avg" : 3.10190308469375E10 }, "totalScheduledTime" : "58.50s", "totalCpuTime" : "16.52s", "totalBlockedTime" : "7.23m", "fullyBlocked" : true, "blockedReasons" : [ ], "physicalInputDataSize" : "0B", "physicalInputPositions" : 0, "physicalInputReadTime" : "0.00ns", "internalNetworkInputDataSize" : "2110724193B", "internalNetworkInputPositions" : 5197502, "rawInputDataSize" : "2110724193B", "rawInputPositions" : 5197502, "processedInputDataSize" : "2374995020B", "processedInputPositions" : 5197502, "outputDataSize" : "0B", "outputPositions" : 0, "physicalWrittenDataSize" : "0B", "operatorSummaries" : [ { "stageId" : 1, "pipelineId" : 0, "operatorId" : 0, "planNodeId" : "165", "operatorType" : "ExchangeOperator", "totalDrivers" : 16, "addInputCalls" : 0, "addInputWall" : "0.00ns", "addInputCpu" : "0.00ns", "physicalInputDataSize" : "0B", "physicalInputPositions" : 0, "internalNetworkInputDataSize" : "2110724193B", "internalNetworkInputPositions" : 5197502, "rawInputDataSize" : "2110724193B", "inputDataSize" : "2374995020B", "inputPositions" : 5197502, "sumSquaredInputPositions" : 1.695206720428E12, "getOutputCalls" : 5308, "getOutputWall" : "42.85s", "getOutputCpu" : "11.51s", "outputDataSize" : "2374995020B", "outputPositions" : 5197502, "physicalWrittenDataSize" : "0B", "blockedWall" : "7.22m", "finishCalls" : 0, "finishWall" : "0.00ns", "finishCpu" : "0.00ns", "userMemoryReservation" : "0B", "revocableMemoryReservation" : "0B", "systemMemoryReservation" : "0B", "peakUserMemoryReservation" : "0B", "peakSystemMemoryReservation" : "0B", "peakRevocableMemoryReservation" : "0B", "peakTotalMemoryReservation" : "0B", "spilledDataSize" : "0B", "info" : { "@type" : "exchangeClientStatus", "bufferedBytes" : 0, "maxBufferedBytes" : 12581962, "averageBytesPerRequest" : 5704461, "successfulRequestsCount" : 5920, "bufferedPages" : 0, "noMoreLocations" : true, "pageBufferClientStatuses" : [ ] } }, { "stageId" : 1, "pipelineId" : 0, "operatorId" : 1, "planNodeId" : "167", "operatorType" : "OrderByOperator", "totalDrivers" : 16, "addInputCalls" : 5214, "addInputWall" : "13.95s", "addInputCpu" : "4.53s", "physicalInputDataSize" : "0B", "physicalInputPositions" : 0, "internalNetworkInputDataSize" : "0B", "internalNetworkInputPositions" : 0, "rawInputDataSize" : "0B", "inputDataSize" : "2374995020B", "inputPositions" : 5197502, "sumSquaredInputPositions" : 1.695206720428E12, "getOutputCalls" : 10924, "getOutputWall" : "1.42s", "getOutputCpu" : "297.01ms", "outputDataSize" : "0B", "outputPositions" : 0, "physicalWrittenDataSize" : "0B", "blockedWall" : "0.00ns", "finishCalls" : 0, "finishWall" : "0.00ns", "finishCpu" : "0.00ns", "userMemoryReservation" : "53678376821B", "revocableMemoryReservation" : "0B", "systemMemoryReservation" : "0B", "peakUserMemoryReservation" : "3783289414B", "peakSystemMemoryReservation" : "0B", "peakRevocableMemoryReservation" : "0B", "peakTotalMemoryReservation" : "3783289414B", "spilledDataSize" : "0B" }, { "stageId" : 1, "pipelineId" : 0, "operatorId" : 2, "planNodeId" : "204", "operatorType" : "LocalExchangeSinkOperator", "totalDrivers" : 16, "addInputCalls" : 0, "addInputWall" : "0.00ns", "addInputCpu" : "0.00ns", "physicalInputDataSize" : "0B", "physicalInputPositions" : 0, "internalNetworkInputDataSize" : "0B", "internalNetworkInputPositions" : 0, "rawInputDataSize" : "0B", "inputDataSize" : "0B", "inputPositions" : 0, "sumSquaredInputPositions" : 0.0, "getOutputCalls" : 0, "getOutputWall" : "0.00ns", "getOutputCpu" : "0.00ns", "outputDataSize" : "0B", "outputPositions" : 0, "physicalWrittenDataSize" : "0B", "blockedWall" : "0.00ns", "finishCalls" : 0, "finishWall" : "0.00ns", "finishCpu" : "0.00ns", "userMemoryReservation" : "0B", "revocableMemoryReservation" : "0B", "systemMemoryReservation" : "0B", "peakUserMemoryReservation" : "0B", "peakSystemMemoryReservation" : "0B", "peakRevocableMemoryReservation" : "0B", "peakTotalMemoryReservation" : "0B", "spilledDataSize" : "0B" } ], "drivers" : [ ] }, { "pipelineId" : 1, "firstStartTime" : "2020-03-16T14:31:53.249Z", "lastStartTime" : "2020-03-16T14:31:53.249Z", "lastEndTime" : "2020-03-16T14:32:24.259Z", "inputPipeline" : false, "outputPipeline" : true, "totalDrivers" : 1, "queuedDrivers" : 0, "queuedPartitionedDrivers" : 0, "runningDrivers" : 0, "runningPartitionedDrivers" : 0, "blockedDrivers" : 1, "completedDrivers" : 0, "userMemoryReservation" : "0B", "revocableMemoryReservation" : "0B", "systemMemoryReservation" : "0B", "queuedTime" : { "count" : 1.0, "total" : 1.0475219E7, "p01" : 1.0475219E7, "p05" : 1.0475219E7, "p10" : 1.0475219E7, "p25" : 1.0475219E7, "p50" : 1.0475219E7, "p75" : 1.0475219E7, "p90" : 1.0475219E7, "p95" : 1.0475219E7, "p99" : 1.0475219E7, "min" : 1.0475219E7, "max" : 1.0475219E7, "avg" : 1.0475219E7 }, "elapsedTime" : { "count" : 1.0, "total" : 3.10198193E10, "p01" : 3.10198193E10, "p05" : 3.10198193E10, "p10" : 3.10198193E10, "p25" : 3.10198193E10, "p50" : 3.10198193E10, "p75" : 3.10198193E10, "p90" : 3.10198193E10, "p95" : 3.10198193E10, "p99" : 3.10198193E10, "min" : 3.10198193E10, "max" : 3.10198193E10, "avg" : 3.10198193E10 }, "totalScheduledTime" : "1.73ms", "totalCpuTime" : "297.00us", "totalBlockedTime" : "31.01s", "fullyBlocked" : true, "blockedReasons" : [ ], "physicalInputDataSize" : "0B", "physicalInputPositions" : 0, "physicalInputReadTime" : "0.00ns", "internalNetworkInputDataSize" : "0B", "internalNetworkInputPositions" : 0, "rawInputDataSize" : "0B", "rawInputPositions" : 0, "processedInputDataSize" : "0B", "processedInputPositions" : 0, "outputDataSize" : "0B", "outputPositions" : 0, "physicalWrittenDataSize" : "0B", "operatorSummaries" : [ { "stageId" : 1, "pipelineId" : 1, "operatorId" : 0, "planNodeId" : "204", "operatorType" : "LocalMergeSourceOperator", "totalDrivers" : 1, "addInputCalls" : 0, "addInputWall" : "0.00ns", "addInputCpu" : "0.00ns", "physicalInputDataSize" : "0B", "physicalInputPositions" : 0, "internalNetworkInputDataSize" : "0B", "internalNetworkInputPositions" : 0, "rawInputDataSize" : "0B", "inputDataSize" : "0B", "inputPositions" : 0, "sumSquaredInputPositions" : 0.0, "getOutputCalls" : 1, "getOutputWall" : "1.67ms", "getOutputCpu" : "234.00us", "outputDataSize" : "0B", "outputPositions" : 0, "physicalWrittenDataSize" : "0B", "blockedWall" : "0.00ns", "finishCalls" : 0, "finishWall" : "0.00ns", "finishCpu" : "0.00ns", "userMemoryReservation" : "0B", "revocableMemoryReservation" : "0B", "systemMemoryReservation" : "0B", "peakUserMemoryReservation" : "0B", "peakSystemMemoryReservation" : "0B", "peakRevocableMemoryReservation" : "0B", "peakTotalMemoryReservation" : "0B", "spilledDataSize" : "0B" }, { "stageId" : 1, "pipelineId" : 1, "operatorId" : 1, "planNodeId" : "204", "operatorType" : "TaskOutputOperator", "totalDrivers" : 1, "addInputCalls" : 0, "addInputWall" : "0.00ns", "addInputCpu" : "0.00ns", "physicalInputDataSize" : "0B", "physicalInputPositions" : 0, "internalNetworkInputDataSize" : "0B", "internalNetworkInputPositions" : 0, "rawInputDataSize" : "0B", "inputDataSize" : "0B", "inputPositions" : 0, "sumSquaredInputPositions" : 0.0, "getOutputCalls" : 0, "getOutputWall" : "0.00ns", "getOutputCpu" : "0.00ns", "outputDataSize" : "0B", "outputPositions" : 0, "physicalWrittenDataSize" : "0B", "blockedWall" : "0.00ns", "finishCalls" : 0, "finishWall" : "0.00ns", "finishCpu" : "0.00ns", "userMemoryReservation" : "0B", "revocableMemoryReservation" : "0B", "systemMemoryReservation" : "0B", "peakUserMemoryReservation" : "0B", "peakSystemMemoryReservation" : "0B", "peakRevocableMemoryReservation" : "0B", "peakTotalMemoryReservation" : "0B", "spilledDataSize" : "0B" } ], "drivers" : [ ] } ] }, "needsPlan" : false } ], "subStages" : [ { "stageId" : "20200316_143152_00013_fbfw4.2", "state" : "FAILED", "plan" : { "id" : "2", "root" : { "@type" : "project", "id" : "2", "source" : { "@type" : "tablescan", "id" : "0", "table" : { "catalogName" : "hive", "connectorHandle" : { "@type" : "hive", "schemaName" : "default", "tableName" : "testorc", "partitionColumns" : [ ], "compactEffectivePredicate" : { "columnDomains" : [ ] }, "enforcedConstraint" : { "columnDomains" : [ ] } }, "transaction" : { "@type" : "hive", "uuid" : "7daebc61-1148-4b7e-b717-00162d290afa" } }, "outputSymbols" : [ "salesforce_contact_id", "email_id", "campaign_id", "sfmc_activity_id", "campaign_name", "sfmc_activity_type", "sfmc_activity_sub_type", "day", "is_delivered", "is_open", "is_inferred_open", "total_opens", "is_link_clicked", "is_hard_bounce", "is_bounce", "is_opt_out", "is_sent", "gl_lob", "region", "subregion", "company_id_l1", "tier", "sales_channel", "channel_name", "primary_topics", "secondary_topics", "subs_frequency", "url_clicked_personalized", "url_alias", "territory_code", "territory_code_0", "territory_code_1", "territory_code_2", "territory_code_3", "country_full", "email_cost", "web_engaged_sub", "email_engaged_sub", "mps_flag", "opportunities", "is_sub", "mpc_flag", "downloads", "video_views", "logins", "pageviews", "guide_to_markets_interactions", "market_insights_email_signups", "registrations", "podcast_plays", "bond_ladder_tool_analysis", "portfolio_insights_tool_analysis", "investment_comparison_tool_analysis", "target_date_compass_tool_analysis", "bond_ladder_tool_report", "portfolio_insights_tool_report", "investment_comparison_tool_report", "target_date_compass_tool_report", "total_tool_report", "total_tool_analysis", "total" ], "assignments" : { "total_tool_report" : { "@type" : "hive", "name" : "total_tool_report", "hiveType" : "double", "type" : "double", "hiveColumnIndex" : 58, "columnType" : "REGULAR" }, "email_cost" : { "@type" : "hive", "name" : "email_cost", "hiveType" : "decimal(2,1)", "type" : "decimal(2,1)", "hiveColumnIndex" : 35, "columnType" : "REGULAR" }, "gl_lob" : { "@type" : "hive", "name" : "gl_lob", "hiveType" : "string", "type" : "varchar", "hiveColumnIndex" : 17, "columnType" : "REGULAR" }, "is_sub" : { "@type" : "hive", "name" : "is_sub", "hiveType" : "double", "type" : "double", "hiveColumnIndex" : 40, "columnType" : "REGULAR" }, "subs_frequency" : { "@type" : "hive", "name" : "subs_frequency", "hiveType" : "varchar(13)", "type" : "varchar(13)", "hiveColumnIndex" : 26, "columnType" : "REGULAR" }, "territory_code_3" : { "@type" : "hive", "name" : "territory_code_5", "hiveType" : "string", "type" : "varchar", "hiveColumnIndex" : 33, "columnType" : "REGULAR" }, "territory_code_1" : { "@type" : "hive", "name" : "territory_code_3", "hiveType" : "string", "type" : "varchar", "hiveColumnIndex" : 31, "columnType" : "REGULAR" }, "secondary_topics" : { "@type" : "hive", "name" : "secondary_topics", "hiveType" : "string", "type" : "varchar", "hiveColumnIndex" : 25, "columnType" : "REGULAR" }, "territory_code_2" : { "@type" : "hive", "name" : "territory_code_4", "hiveType" : "string", "type" : "varchar", "hiveColumnIndex" : 32, "columnType" : "REGULAR" }, "is_bounce" : { "@type" : "hive", "name" : "is_bounce", "hiveType" : "double", "type" : "double", "hiveColumnIndex" : 14, "columnType" : "REGULAR" }, "territory_code_0" : { "@type" : "hive", "name" : "territory_code_2", "hiveType" : "string", "type" : "varchar", "hiveColumnIndex" : 30, "columnType" : "REGULAR" }, "primary_topics" : { "@type" : "hive", "name" : "primary_topics", "hiveType" : "string", "type" : "varchar", "hiveColumnIndex" : 24, "columnType" : "REGULAR" }, "is_sent" : { "@type" : "hive", "name" : "is_sent", "hiveType" : "double", "type" : "double", "hiveColumnIndex" : 16, "columnType" : "REGULAR" }, "video_views" : { "@type" : "hive", "name" : "video_views", "hiveType" : "double", "type" : "double", "hiveColumnIndex" : 43, "columnType" : "REGULAR" }, "total_tool_analysis" : { "@type" : "hive", "name" : "total_tool_analysis", "hiveType" : "double", "type" : "double", "hiveColumnIndex" : 59, "columnType" : "REGULAR" }, "day" : { "@type" : "hive", "name" : "day", "hiveType" : "date", "type" : "date", "hiveColumnIndex" : 7, "columnType" : "REGULAR" }, "investment_comparison_tool_report" : { "@type" : "hive", "name" : "investment_comparison_tool_report", "hiveType" : "double", "type" : "double", "hiveColumnIndex" : 56, "columnType" : "REGULAR" }, "market_insights_email_signups" : { "@type" : "hive", "name" : "market_insights_email_signups", "hiveType" : "double", "type" : "double", "hiveColumnIndex" : 47, "columnType" : "REGULAR" }, "campaign_id" : { "@type" : "hive", "name" : "campaign_id", "hiveType" : "string", "type" : "varchar", "hiveColumnIndex" : 2, "columnType" : "REGULAR" }, "territory_code" : { "@type" : "hive", "name" : "territory_code_1", "hiveType" : "string", "type" : "varchar", "hiveColumnIndex" : 29, "columnType" : "REGULAR" }, "registrations" : { "@type" : "hive", "name" : "registrations", "hiveType" : "double", "type" : "double", "hiveColumnIndex" : 48, "columnType" : "REGULAR" }, "is_open" : { "@type" : "hive", "name" : "is_open", "hiveType" : "double", "type" : "double", "hiveColumnIndex" : 9, "columnType" : "REGULAR" }, "subregion" : { "@type" : "hive", "name" : "subregion", "hiveType" : "string", "type" : "varchar", "hiveColumnIndex" : 19, "columnType" : "REGULAR" }, "sales_channel" : { "@type" : "hive", "name" : "sales_channel", "hiveType" : "string", "type" : "varchar", "hiveColumnIndex" : 22, "columnType" : "REGULAR" }, "is_hard_bounce" : { "@type" : "hive", "name" : "is_hard_bounce", "hiveType" : "double", "type" : "double", "hiveColumnIndex" : 13, "columnType" : "REGULAR" }, "podcast_plays" : { "@type" : "hive", "name" : "podcast_plays", "hiveType" : "double", "type" : "double", "hiveColumnIndex" : 49, "columnType" : "REGULAR" }, "is_inferred_open" : { "@type" : "hive", "name" : "is_inferred_open", "hiveType" : "double", "type" : "double", "hiveColumnIndex" : 10, "columnType" : "REGULAR" }, "salesforce_contact_id" : { "@type" : "hive", "name" : "salesforce_contact_id", "hiveType" : "string", "type" : "varchar", "hiveColumnIndex" : 0, "columnType" : "REGULAR" }, "target_date_compass_tool_analysis" : { "@type" : "hive", "name" : "target_date_compass_tool_analysis", "hiveType" : "double", "type" : "double", "hiveColumnIndex" : 53, "columnType" : "REGULAR" }, "email_engaged_sub" : { "@type" : "hive", "name" : "email_engaged_sub", "hiveType" : "decimal(2,1)", "type" : "decimal(2,1)", "hiveColumnIndex" : 37, "columnType" : "REGULAR" }, "region" : { "@type" : "hive", "name" : "region", "hiveType" : "string", "type" : "varchar", "hiveColumnIndex" : 18, "columnType" : "REGULAR" }, "logins" : { "@type" : "hive", "name" : "logins", "hiveType" : "double", "type" : "double", "hiveColumnIndex" : 44, "columnType" : "REGULAR" }, "email_id" : { "@type" : "hive", "name" : "email_id", "hiveType" : "string", "type" : "varchar", "hiveColumnIndex" : 1, "columnType" : "REGULAR" }, "is_opt_out" : { "@type" : "hive", "name" : "is_opt_out", "hiveType" : "double", "type" : "double", "hiveColumnIndex" : 15, "columnType" : "REGULAR" }, "is_delivered" : { "@type" : "hive", "name" : "is_delivered", "hiveType" : "double", "type" : "double", "hiveColumnIndex" : 8, "columnType" : "REGULAR" }, "investment_comparison_tool_analysis" : { "@type" : "hive", "name" : "investment_comparison_tool_analysis", "hiveType" : "double", "type" : "double", "hiveColumnIndex" : 52, "columnType" : "REGULAR" }, "portfolio_insights_tool_analysis" : { "@type" : "hive", "name" : "portfolio_insights_tool_analysis", "hiveType" : "double", "type" : "double", "hiveColumnIndex" : 51, "columnType" : "REGULAR" }, "country_full" : { "@type" : "hive", "name" : "country_full", "hiveType" : "string", "type" : "varchar", "hiveColumnIndex" : 34, "columnType" : "REGULAR" }, "campaign_name" : { "@type" : "hive", "name" : "campaign_name", "hiveType" : "string", "type" : "varchar", "hiveColumnIndex" : 4, "columnType" : "REGULAR" }, "total" : { "@type" : "hive", "name" : "total", "hiveType" : "double", "type" : "double", "hiveColumnIndex" : 60, "columnType" : "REGULAR" }, "pageviews" : { "@type" : "hive", "name" : "pageviews", "hiveType" : "double", "type" : "double", "hiveColumnIndex" : 45, "columnType" : "REGULAR" }, "url_alias" : { "@type" : "hive", "name" : "url_alias_80", "hiveType" : "string", "type" : "varchar", "hiveColumnIndex" : 28, "columnType" : "REGULAR" }, "bond_ladder_tool_analysis" : { "@type" : "hive", "name" : "bond_ladder_tool_analysis", "hiveType" : "double", "type" : "double", "hiveColumnIndex" : 50, "columnType" : "REGULAR" }, "tier" : { "@type" : "hive", "name" : "tier", "hiveType" : "string", "type" : "varchar", "hiveColumnIndex" : 21, "columnType" : "REGULAR" }, "downloads" : { "@type" : "hive", "name" : "downloads", "hiveType" : "double", "type" : "double", "hiveColumnIndex" : 42, "columnType" : "REGULAR" }, "is_link_clicked" : { "@type" : "hive", "name" : "is_link_clicked", "hiveType" : "double", "type" : "double", "hiveColumnIndex" : 12, "columnType" : "REGULAR" }, "web_engaged_sub" : { "@type" : "hive", "name" : "web_engaged_sub", "hiveType" : "decimal(2,1)", "type" : "decimal(2,1)", "hiveColumnIndex" : 36, "columnType" : "REGULAR" }, "mps_flag" : { "@type" : "hive", "name" : "mps_flag", "hiveType" : "decimal(2,1)", "type" : "decimal(2,1)", "hiveColumnIndex" : 38, "columnType" : "REGULAR" }, "url_clicked_personalized" : { "@type" : "hive", "name" : "url_clicked_personalized", "hiveType" : "string", "type" : "varchar", "hiveColumnIndex" : 27, "columnType" : "REGULAR" }, "sfmc_activity_sub_type" : { "@type" : "hive", "name" : "sfmc_activity_sub_type", "hiveType" : "string", "type" : "varchar", "hiveColumnIndex" : 6, "columnType" : "REGULAR" }, "mpc_flag" : { "@type" : "hive", "name" : "mpc_flag", "hiveType" : "int", "type" : "integer", "hiveColumnIndex" : 41, "columnType" : "REGULAR" }, "portfolio_insights_tool_report" : { "@type" : "hive", "name" : "portfolio_insights_tool_report", "hiveType" : "double", "type" : "double", "hiveColumnIndex" : 55, "columnType" : "REGULAR" }, "channel_name" : { "@type" : "hive", "name" : "channel_name", "hiveType" : "string", "type" : "varchar", "hiveColumnIndex" : 23, "columnType" : "REGULAR" }, "guide_to_markets_interactions" : { "@type" : "hive", "name" : "guide_to_markets_interactions", "hiveType" : "double", "type" : "double", "hiveColumnIndex" : 46, "columnType" : "REGULAR" }, "sfmc_activity_type" : { "@type" : "hive", "name" : "sfmc_activity_type", "hiveType" : "string", "type" : "varchar", "hiveColumnIndex" : 5, "columnType" : "REGULAR" }, "opportunities" : { "@type" : "hive", "name" : "opportunities", "hiveType" : "bigint", "type" : "bigint", "hiveColumnIndex" : 39, "columnType" : "REGULAR" }, "company_id_l1" : { "@type" : "hive", "name" : "company_id_l1", "hiveType" : "string", "type" : "varchar", "hiveColumnIndex" : 20, "columnType" : "REGULAR" }, "total_opens" : { "@type" : "hive", "name" : "total_opens", "hiveType" : "double", "type" : "double", "hiveColumnIndex" : 11, "columnType" : "REGULAR" }, "sfmc_activity_id" : { "@type" : "hive", "name" : "sfmc_activity_id", "hiveType" : "string", "type" : "varchar", "hiveColumnIndex" : 3, "columnType" : "REGULAR" }, "bond_ladder_tool_report" : { "@type" : "hive", "name" : "bond_ladder_tool_report", "hiveType" : "double", "type" : "double", "hiveColumnIndex" : 54, "columnType" : "REGULAR" }, "target_date_compass_tool_report" : { "@type" : "hive", "name" : "target_date_compass_tool_report", "hiveType" : "double", "type" : "double", "hiveColumnIndex" : 57, "columnType" : "REGULAR" } } }, "assignments" : { "assignments" : { "random" : "\"@random|double@random():double\"()", "salesforce_contact_id" : "\"salesforce_contact_id\"", "email_id" : "\"email_id\"", "campaign_id" : "\"campaign_id\"", "sfmc_activity_id" : "\"sfmc_activity_id\"", "campaign_name" : "\"campaign_name\"", "sfmc_activity_type" : "\"sfmc_activity_type\"", "sfmc_activity_sub_type" : "\"sfmc_activity_sub_type\"", "day" : "\"day\"", "is_delivered" : "\"is_delivered\"", "is_open" : "\"is_open\"", "is_inferred_open" : "\"is_inferred_open\"", "total_opens" : "\"total_opens\"", "is_link_clicked" : "\"is_link_clicked\"", "is_hard_bounce" : "\"is_hard_bounce\"", "is_bounce" : "\"is_bounce\"", "is_opt_out" : "\"is_opt_out\"", "is_sent" : "\"is_sent\"", "gl_lob" : "\"gl_lob\"", "region" : "\"region\"", "subregion" : "\"subregion\"", "company_id_l1" : "\"company_id_l1\"", "tier" : "\"tier\"", "sales_channel" : "\"sales_channel\"", "channel_name" : "\"channel_name\"", "primary_topics" : "\"primary_topics\"", "secondary_topics" : "\"secondary_topics\"", "subs_frequency" : "\"subs_frequency\"", "url_clicked_personalized" : "\"url_clicked_personalized\"", "url_alias" : "\"url_alias\"", "territory_code" : "\"territory_code\"", "territory_code_0" : "\"territory_code_0\"", "territory_code_1" : "\"territory_code_1\"", "territory_code_2" : "\"territory_code_2\"", "territory_code_3" : "\"territory_code_3\"", "country_full" : "\"country_full\"", "email_cost" : "\"email_cost\"", "web_engaged_sub" : "\"web_engaged_sub\"", "email_engaged_sub" : "\"email_engaged_sub\"", "mps_flag" : "\"mps_flag\"", "opportunities" : "\"opportunities\"", "is_sub" : "\"is_sub\"", "mpc_flag" : "\"mpc_flag\"", "downloads" : "\"downloads\"", "video_views" : "\"video_views\"", "logins" : "\"logins\"", "pageviews" : "\"pageviews\"", "guide_to_markets_interactions" : "\"guide_to_markets_interactions\"", "market_insights_email_signups" : "\"market_insights_email_signups\"", "registrations" : "\"registrations\"", "podcast_plays" : "\"podcast_plays\"", "bond_ladder_tool_analysis" : "\"bond_ladder_tool_analysis\"", "portfolio_insights_tool_analysis" : "\"portfolio_insights_tool_analysis\"", "investment_comparison_tool_analysis" : "\"investment_comparison_tool_analysis\"", "target_date_compass_tool_analysis" : "\"target_date_compass_tool_analysis\"", "bond_ladder_tool_report" : "\"bond_ladder_tool_report\"", "portfolio_insights_tool_report" : "\"portfolio_insights_tool_report\"", "investment_comparison_tool_report" : "\"investment_comparison_tool_report\"", "target_date_compass_tool_report" : "\"target_date_compass_tool_report\"", "total_tool_report" : "\"total_tool_report\"", "total_tool_analysis" : "\"total_tool_analysis\"", "total" : "\"total\"" } } }, "symbols" : { "email_cost" : "decimal(2,1)", "subs_frequency" : "varchar(13)", "is_sub" : "double", "is_bounce" : "double", "primary_topics" : "varchar", "investment_comparison_tool_report" : "double", "registrations" : "double", "is_open" : "double", "subregion" : "varchar", "bond_ladder_tool_analysis" : "double", "web_engaged_sub" : "decimal(2,1)", "mps_flag" : "decimal(2,1)", "sfmc_activity_sub_type" : "varchar", "sfmc_activity_type" : "varchar", "sfmc_activity_id" : "varchar", "bond_ladder_tool_report" : "double", "total_tool_report" : "double", "territory_code_3" : "varchar", "secondary_topics" : "varchar", "territory_code_1" : "varchar", "territory_code_2" : "varchar", "territory_code_0" : "varchar", "total_tool_analysis" : "double", "campaign_id" : "varchar", "is_inferred_open" : "double", "salesforce_contact_id" : "varchar", "region" : "varchar", "logins" : "double", "is_delivered" : "double", "random" : "double", "pageviews" : "double", "downloads" : "double", "is_link_clicked" : "double", "mpc_flag" : "integer", "portfolio_insights_tool_report" : "double", "total_opens" : "double", "target_date_compass_tool_report" : "double", "gl_lob" : "varchar", "video_views" : "double", "sales_channel" : "varchar", "is_hard_bounce" : "double", "podcast_plays" : "double", "target_date_compass_tool_analysis" : "double", "is_opt_out" : "double", "investment_comparison_tool_analysis" : "double", "channel_name" : "varchar", "opportunities" : "bigint", "company_id_l1" : "varchar", "is_sent" : "double", "day" : "date", "market_insights_email_signups" : "double", "territory_code" : "varchar", "email_engaged_sub" : "decimal(2,1)", "email_id" : "varchar", "portfolio_insights_tool_analysis" : "double", "country_full" : "varchar", "campaign_name" : "varchar", "total" : "double", "url_alias" : "varchar", "tier" : "varchar", "url_clicked_personalized" : "varchar", "guide_to_markets_interactions" : "double" }, "partitioning" : { "connectorHandle" : { "@type" : "$remote", "partitioning" : "SOURCE", "function" : "UNKNOWN" } }, "partitionedSources" : [ "0" ], "partitioningScheme" : { "partitioning" : { "handle" : { "connectorHandle" : { "@type" : "$remote", "partitioning" : "FIXED", "function" : "ROUND_ROBIN" } }, "arguments" : [ ] }, "outputLayout" : [ "random", "salesforce_contact_id", "email_id", "campaign_id", "sfmc_activity_id", "campaign_name", "sfmc_activity_type", "sfmc_activity_sub_type", "day", "is_delivered", "is_open", "is_inferred_open", "total_opens", "is_link_clicked", "is_hard_bounce", "is_bounce", "is_opt_out", "is_sent", "gl_lob", "region", "subregion", "company_id_l1", "tier", "sales_channel", "channel_name", "primary_topics", "secondary_topics", "subs_frequency", "url_clicked_personalized", "url_alias", "territory_code", "territory_code_0", "territory_code_1", "territory_code_2", "territory_code_3", "country_full", "email_cost", "web_engaged_sub", "email_engaged_sub", "mps_flag", "opportunities", "is_sub", "mpc_flag", "downloads", "video_views", "logins", "pageviews", "guide_to_markets_interactions", "market_insights_email_signups", "registrations", "podcast_plays", "bond_ladder_tool_analysis", "portfolio_insights_tool_analysis", "investment_comparison_tool_analysis", "target_date_compass_tool_analysis", "bond_ladder_tool_report", "portfolio_insights_tool_report", "investment_comparison_tool_report", "target_date_compass_tool_report", "total_tool_report", "total_tool_analysis", "total" ], "replicateNullsAndAny" : false, "bucketToPartition" : [ 0 ] }, "stageExecutionDescriptor" : { "strategy" : "UNGROUPED_EXECUTION", "groupedExecutionScanNodes" : [ ] }, "statsAndCosts" : { "stats" : { }, "costs" : { } }, "jsonRepresentation" : "{\n \"id\" : \"2\",\n \"name\" : \"ScanProject\",\n \"identifier\" : \"[table = hive:default:testorc]\",\n \"details\" : \"random := random()\ntotal_tool_report := total_tool_report:double:58:REGULAR\nemail_cost := email_cost:decimal(2,1):35:REGULAR\ngl_lob := gl_lob:string:17:REGULAR\nis_sub := is_sub:double:40:REGULAR\nsubs_frequency := subs_frequency:varchar(13):26:REGULAR\nterritory_code_3 := territory_code_5:string:33:REGULAR\nterritory_code_1 := territory_code_3:string:31:REGULAR\nsecondary_topics := secondary_topics:string:25:REGULAR\nterritory_code_2 := territory_code_4:string:32:REGULAR\nis_bounce := is_bounce:double:14:REGULAR\nterritory_code_0 := territory_code_2:string:30:REGULAR\nprimary_topics := primary_topics:string:24:REGULAR\nis_sent := is_sent:double:16:REGULAR\nvideo_views := video_views:double:43:REGULAR\ntotal_tool_analysis := total_tool_analysis:double:59:REGULAR\nday := day:date:7:REGULAR\ninvestment_comparison_tool_report := investment_comparison_tool_report:double:56:REGULAR\nmarket_insights_email_signups := market_insights_email_signups:double:47:REGULAR\ncampaign_id := campaign_id:string:2:REGULAR\nterritory_code := territory_code_1:string:29:REGULAR\nregistrations := registrations:double:48:REGULAR\nis_open := is_open:double:9:REGULAR\nsubregion := subregion:string:19:REGULAR\nsales_channel := sales_channel:string:22:REGULAR\nis_hard_bounce := is_hard_bounce:double:13:REGULAR\npodcast_plays := podcast_plays:double:49:REGULAR\nis_inferred_open := is_inferred_open:double:10:REGULAR\nsalesforce_contact_id := salesforce_contact_id:string:0:REGULAR\ntarget_date_compass_tool_analysis := target_date_compass_tool_analysis:double:53:REGULAR\nemail_engaged_sub := email_engaged_sub:decimal(2,1):37:REGULAR\nregion := region:string:18:REGULAR\nlogins := logins:double:44:REGULAR\nemail_id := email_id:string:1:REGULAR\nis_opt_out := is_opt_out:double:15:REGULAR\nis_delivered := is_delivered:double:8:REGULAR\ninvestment_comparison_tool_analysis := investment_comparison_tool_analysis:double:52:REGULAR\nportfolio_insights_tool_analysis := portfolio_insights_tool_analysis:double:51:REGULAR\ncountry_full := country_full:string:34:REGULAR\ncampaign_name := campaign_name:string:4:REGULAR\ntotal := total:double:60:REGULAR\npageviews := pageviews:double:45:REGULAR\nurl_alias := url_alias_80:string:28:REGULAR\nbond_ladder_tool_analysis := bond_ladder_tool_analysis:double:50:REGULAR\ntier := tier:string:21:REGULAR\ndownloads := downloads:double:42:REGULAR\nis_link_clicked := is_link_clicked:double:12:REGULAR\nweb_engaged_sub := web_engaged_sub:decimal(2,1):36:REGULAR\nmps_flag := mps_flag:decimal(2,1):38:REGULAR\nurl_clicked_personalized := url_clicked_personalized:string:27:REGULAR\nsfmc_activity_sub_type := sfmc_activity_sub_type:string:6:REGULAR\nmpc_flag := mpc_flag:int:41:REGULAR\nportfolio_insights_tool_report := portfolio_insights_tool_report:double:55:REGULAR\nchannel_name := channel_name:string:23:REGULAR\nguide_to_markets_interactions := guide_to_markets_interactions:double:46:REGULAR\nsfmc_activity_type := sfmc_activity_type:string:5:REGULAR\nopportunities := opportunities:bigint:39:REGULAR\ncompany_id_l1 := company_id_l1:string:20:REGULAR\ntotal_opens := total_opens:double:11:REGULAR\nsfmc_activity_id := sfmc_activity_id:string:3:REGULAR\nbond_ladder_tool_report := bond_ladder_tool_report:double:54:REGULAR\ntarget_date_compass_tool_report := target_date_compass_tool_report:double:57:REGULAR\n\",\n \"children\" : [ ],\n \"remoteSources\" : [ ]\n}" }, "types" : [ "double", "varchar", "varchar", "varchar", "varchar", "varchar", "varchar", "varchar", "date", "double", "double", "double", "double", "double", "double", "double", "double", "double", "varchar", "varchar", "varchar", "varchar", "varchar", "varchar", "varchar", "varchar", "varchar", "varchar(13)", "varchar", "varchar", "varchar", "varchar", "varchar", "varchar", "varchar", "varchar", "decimal(2,1)", "decimal(2,1)", "decimal(2,1)", "decimal(2,1)", "bigint", "double", "integer", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double", "double" ], "stageStats" : { "schedulingComplete" : "2020-03-16T14:31:53.510Z", "getSplitDistribution" : { "count" : 1.0, "total" : 3.21233123E8, "p01" : 3.21233123E8, "p05" : 3.21233123E8, "p10" : 3.21233123E8, "p25" : 3.21233123E8, "p50" : 3.21233123E8, "p75" : 3.21233123E8, "p90" : 3.21233123E8, "p95" : 3.21233123E8, "p99" : 3.21233123E8, "min" : 3.21233123E8, "max" : 3.21233123E8, "avg" : 3.21233123E8 }, "totalTasks" : 1, "runningTasks" : 0, "completedTasks" : 1, "totalDrivers" : 6, "queuedDrivers" : 0, "runningDrivers" : 1, "blockedDrivers" : 0, "completedDrivers" : 5, "cumulativeUserMemory" : 0.0, "userMemoryReservation" : "0B", "revocableMemoryReservation" : "0B", "totalMemoryReservation" : "18937313B", "peakUserMemoryReservation" : "0B", "peakRevocableMemoryReservation" : "0B", "totalScheduledTime" : "2.20m", "totalCpuTime" : "55.05s", "totalBlockedTime" : "22.17s", "fullyBlocked" : false, "blockedReasons" : [ ], "physicalInputDataSize" : "181576997B", "physicalInputPositions" : 5198526, "physicalInputReadTime" : "1.10s", "internalNetworkInputDataSize" : "0B", "internalNetworkInputPositions" : 0, "rawInputDataSize" : "181576997B", "rawInputPositions" : 5198526, "processedInputDataSize" : "2327849012B", "processedInputPositions" : 5198526, "bufferedDataSize" : "0B", "outputDataSize" : "2374995020B", "outputPositions" : 5197502, "physicalWrittenDataSize" : "0B", "gcInfo" : { "stageId" : 2, "tasks" : 1, "fullGcTasks" : 0, "minFullGcSec" : 0, "maxFullGcSec" : 0, "totalFullGcSec" : 0, "averageFullGcSec" : 0 }, "operatorSummaries" : [ { "stageId" : 2, "pipelineId" : 0, "operatorId" : 0, "planNodeId" : "2", "operatorType" : "ScanFilterAndProjectOperator", "totalDrivers" : 6, "addInputCalls" : 5208, "addInputWall" : "1.10s", "addInputCpu" : "0.00ns", "physicalInputDataSize" : "181576997B", "physicalInputPositions" : 5198526, "internalNetworkInputDataSize" : "0B", "internalNetworkInputPositions" : 0, "rawInputDataSize" : "181576997B", "inputDataSize" : "2327849012B", "inputPositions" : 5198526, "sumSquaredInputPositions" : 5.188204009348E12, "getOutputCalls" : 5235, "getOutputWall" : "1.04m", "getOutputCpu" : "26.02s", "outputDataSize" : "2375477893B", "outputPositions" : 5198526, "physicalWrittenDataSize" : "0B", "blockedWall" : "0.00ns", "finishCalls" : 0, "finishWall" : "0.00ns", "finishCpu" : "0.00ns", "userMemoryReservation" : "0B", "revocableMemoryReservation" : "0B", "systemMemoryReservation" : "18937313B", "peakUserMemoryReservation" : "0B", "peakSystemMemoryReservation" : "38629863B", "peakRevocableMemoryReservation" : "0B", "peakTotalMemoryReservation" : "38629863B", "spilledDataSize" : "0B" }, { "stageId" : 2, "pipelineId" : 0, "operatorId" : 1, "planNodeId" : "2", "operatorType" : "TaskOutputOperator", "totalDrivers" : 6, "addInputCalls" : 5214, "addInputWall" : "1.16m", "addInputCpu" : "28.98s", "physicalInputDataSize" : "0B", "physicalInputPositions" : 0, "internalNetworkInputDataSize" : "0B", "internalNetworkInputPositions" : 0, "rawInputDataSize" : "0B", "inputDataSize" : "2374995020B", "inputPositions" : 5197502, "sumSquaredInputPositions" : 5.18595134042E12, "getOutputCalls" : 0, "getOutputWall" : "0.00ns", "getOutputCpu" : "0.00ns", "outputDataSize" : "2374995020B", "outputPositions" : 5197502, "physicalWrittenDataSize" : "0B", "blockedWall" : "22.17s", "finishCalls" : 5, "finishWall" : "74.68us", "finishCpu" : "98.00us", "userMemoryReservation" : "0B", "revocableMemoryReservation" : "0B", "systemMemoryReservation" : "0B", "peakUserMemoryReservation" : "0B", "peakSystemMemoryReservation" : "0B", "peakRevocableMemoryReservation" : "0B", "peakTotalMemoryReservation" : "0B", "spilledDataSize" : "0B" } ] }, "tasks" : [ { "taskStatus" : { "taskId" : "20200316_143152_00013_fbfw4.2.0", "taskInstanceId" : "548c6d6e-44fd-4c34-bf46-5513d3d8a368", "version" : 332, "state" : "FAILED", "self" : "http://localhost:8080/v1/task/20200316_143152_00013_fbfw4.2.0", "nodeId" : "ffffffff-ffff-ffff-ffff-ffffffffffff", "completedDriverGroups" : [ ], "failures" : [ { "type" : "io.prestosql.ExceededMemoryLimitException", "message" : "Query exceeded per-node total memory limit of 50GB [Allocated: 50.01GB, Delta: 429.71kB, Top Consumers: {OrderByOperator=49.99GB, ScanFilterAndProjectOperator=18.06MB}]", "suppressed" : [ ], "stack" : [ "io.prestosql.ExceededMemoryLimitException.exceededLocalTotalMemoryLimit(ExceededMemoryLimitException.java:46)", "io.prestosql.memory.QueryContext.enforceTotalMemoryLimit(QueryContext.java:342)", "io.prestosql.memory.QueryContext.updateSystemMemory(QueryContext.java:175)", "io.prestosql.memory.QueryContext$QueryMemoryReservationHandler.reserveMemory(QueryContext.java:313)", "io.prestosql.memory.context.RootAggregatedMemoryContext.updateBytes(RootAggregatedMemoryContext.java:37)", "io.prestosql.memory.context.ChildAggregatedMemoryContext.updateBytes(ChildAggregatedMemoryContext.java:38)", "io.prestosql.memory.context.SimpleLocalMemoryContext.setBytes(SimpleLocalMemoryContext.java:66)", "io.prestosql.execution.buffer.OutputBufferMemoryManager.updateMemoryUsage(OutputBufferMemoryManager.java:86)", "io.prestosql.execution.buffer.ArbitraryOutputBuffer.enqueue(ArbitraryOutputBuffer.java:220)", "io.prestosql.execution.buffer.LazyOutputBuffer.enqueue(LazyOutputBuffer.java:254)", "io.prestosql.operator.TaskOutputOperator.addInput(TaskOutputOperator.java:149)", "io.prestosql.operator.Driver.processInternal(Driver.java:384)", "io.prestosql.operator.Driver.lambda$processFor$8(Driver.java:283)", "io.prestosql.operator.Driver.tryWithLock(Driver.java:675)", "io.prestosql.operator.Driver.processFor(Driver.java:276)", "io.prestosql.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:1075)", "io.prestosql.execution.executor.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:163)", "io.prestosql.execution.executor.TaskExecutor$TaskRunner.run(TaskExecutor.java:484)", "io.prestosql.$gen.Presto_nulltestversion__20200316_143022_2.run(Unknown Source)", "java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)", "java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)", "java.lang.Thread.run(Thread.java:748)" ], "errorCode" : { "code" : 131079, "name" : "EXCEEDED_LOCAL_MEMORY_LIMIT", "type" : "INSUFFICIENT_RESOURCES" } }, { "type" : "io.prestosql.ExceededMemoryLimitException", "message" : "Query exceeded per-node total memory limit of 50GB [Allocated: 50.01GB, Delta: 429.71kB, Top Consumers: {OrderByOperator=49.99GB, ScanFilterAndProjectOperator=18.06MB}]", "suppressed" : [ ], "stack" : [ "io.prestosql.ExceededMemoryLimitException.exceededLocalTotalMemoryLimit(ExceededMemoryLimitException.java:46)", "io.prestosql.memory.QueryContext.enforceTotalMemoryLimit(QueryContext.java:342)", "io.prestosql.memory.QueryContext.updateSystemMemory(QueryContext.java:175)", "io.prestosql.memory.QueryContext$QueryMemoryReservationHandler.reserveMemory(QueryContext.java:313)", "io.prestosql.memory.context.RootAggregatedMemoryContext.updateBytes(RootAggregatedMemoryContext.java:37)", "io.prestosql.memory.context.ChildAggregatedMemoryContext.updateBytes(ChildAggregatedMemoryContext.java:38)", "io.prestosql.memory.context.SimpleLocalMemoryContext.setBytes(SimpleLocalMemoryContext.java:66)", "io.prestosql.execution.buffer.OutputBufferMemoryManager.updateMemoryUsage(OutputBufferMemoryManager.java:86)", "io.prestosql.execution.buffer.ArbitraryOutputBuffer.enqueue(ArbitraryOutputBuffer.java:220)", "io.prestosql.execution.buffer.LazyOutputBuffer.enqueue(LazyOutputBuffer.java:254)", "io.prestosql.operator.TaskOutputOperator.addInput(TaskOutputOperator.java:149)", "io.prestosql.operator.Driver.processInternal(Driver.java:384)", "io.prestosql.operator.Driver.lambda$processFor$8(Driver.java:283)", "io.prestosql.operator.Driver.tryWithLock(Driver.java:675)", "io.prestosql.operator.Driver.processFor(Driver.java:276)", "io.prestosql.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:1075)", "io.prestosql.execution.executor.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:163)", "io.prestosql.execution.executor.TaskExecutor$TaskRunner.run(TaskExecutor.java:484)", "io.prestosql.$gen.Presto_nulltestversion__20200316_143022_2.run(Unknown Source)", "java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)", "java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)", "java.lang.Thread.run(Thread.java:748)" ], "errorCode" : { "code" : 131079, "name" : "EXCEEDED_LOCAL_MEMORY_LIMIT", "type" : "INSUFFICIENT_RESOURCES" } } ], "queuedPartitionedDrivers" : 0, "runningPartitionedDrivers" : 1, "outputBufferOverutilized" : true, "physicalWrittenDataSize" : "0B", "memoryReservation" : "0B", "systemMemoryReservation" : "18937313B", "revocableMemoryReservation" : "0B", "fullGcCount" : 0, "fullGcTime" : "0.00ns" }, "lastHeartbeat" : "2020-03-16T14:32:24.206Z", "outputBuffers" : { "type" : "ARBITRARY", "state" : "FAILED", "canAddBuffers" : false, "canAddPages" : false, "totalBufferedBytes" : 0, "totalBufferedPages" : 0, "totalRowsSent" : 5197502, "totalPagesSent" : 5214, "buffers" : [ ] }, "noMoreSplits" : [ "0" ], "stats" : { "createTime" : "2020-03-16T14:31:53.510Z", "firstStartTime" : "2020-03-16T14:31:53.611Z", "lastStartTime" : "2020-03-16T14:31:53.614Z", "lastEndTime" : "2020-03-16T14:32:23.974Z", "endTime" : "2020-03-16T14:32:24.215Z", "elapsedTime" : "30.65s", "queuedTime" : "46.62ms", "totalDrivers" : 6, "queuedDrivers" : 0, "queuedPartitionedDrivers" : 0, "runningDrivers" : 1, "runningPartitionedDrivers" : 1, "blockedDrivers" : 0, "completedDrivers" : 5, "cumulativeUserMemory" : 0.0, "userMemoryReservation" : "0B", "revocableMemoryReservation" : "0B", "systemMemoryReservation" : "18937313B", "totalScheduledTime" : "2.20m", "totalCpuTime" : "55.05s", "totalBlockedTime" : "22.17s", "fullyBlocked" : false, "blockedReasons" : [ ], "physicalInputDataSize" : "181576997B", "physicalInputPositions" : 5198526, "physicalInputReadTime" : "1.10s", "internalNetworkInputDataSize" : "0B", "internalNetworkInputPositions" : 0, "rawInputDataSize" : "181576997B", "rawInputPositions" : 5198526, "processedInputDataSize" : "2327849012B", "processedInputPositions" : 5198526, "outputDataSize" : "2374995020B", "outputPositions" : 5197502, "physicalWrittenDataSize" : "0B", "fullGcCount" : 0, "fullGcTime" : "0.00ns", "pipelines" : [ { "pipelineId" : 0, "firstStartTime" : "2020-03-16T14:31:53.611Z", "lastStartTime" : "2020-03-16T14:31:53.614Z", "lastEndTime" : "2020-03-16T14:32:23.974Z", "inputPipeline" : true, "outputPipeline" : true, "totalDrivers" : 6, "queuedDrivers" : 0, "queuedPartitionedDrivers" : 0, "runningDrivers" : 1, "runningPartitionedDrivers" : 1, "blockedDrivers" : 0, "completedDrivers" : 5, "userMemoryReservation" : "0B", "revocableMemoryReservation" : "0B", "systemMemoryReservation" : "18937313B", "queuedTime" : { "count" : 6.0, "total" : 2.9881489E7, "p01" : 3832481.0, "p05" : 3832481.0, "p10" : 3832481.0, "p25" : 4131074.0, "p50" : 5300517.0, "p75" : 5666179.0, "p90" : 6119214.0, "p95" : 6119214.0, "p99" : 6119214.0, "min" : 3832481.0, "max" : 6119214.0, "avg" : 4980248.166666667 }, "elapsedTime" : { "count" : 6.0, "total" : 1.5503674389E11, "p01" : 5.090943339E9, "p05" : 5.090943339E9, "p10" : 5.090943339E9, "p25" : 2.9223281413E10, "p50" : 3.0284489926E10, "p75" : 3.03666747E10, "p90" : 3.0607761625E10, "p95" : 3.0607761625E10, "p99" : 3.0607761625E10, "min" : 5.090943339E9, "max" : 3.0607761625E10, "avg" : 2.5839457315E10 }, "totalScheduledTime" : "2.20m", "totalCpuTime" : "55.05s", "totalBlockedTime" : "22.17s", "fullyBlocked" : false, "blockedReasons" : [ ], "physicalInputDataSize" : "181576997B", "physicalInputPositions" : 5198526, "physicalInputReadTime" : "1.10s", "internalNetworkInputDataSize" : "0B", "internalNetworkInputPositions" : 0, "rawInputDataSize" : "181576997B", "rawInputPositions" : 5198526, "processedInputDataSize" : "2327849012B", "processedInputPositions" : 5198526, "outputDataSize" : "2374995020B", "outputPositions" : 5197502, "physicalWrittenDataSize" : "0B", "operatorSummaries" : [ { "stageId" : 2, "pipelineId" : 0, "operatorId" : 0, "planNodeId" : "2", "operatorType" : "ScanFilterAndProjectOperator", "totalDrivers" : 6, "addInputCalls" : 5208, "addInputWall" : "1.10s", "addInputCpu" : "0.00ns", "physicalInputDataSize" : "181576997B", "physicalInputPositions" : 5198526, "internalNetworkInputDataSize" : "0B", "internalNetworkInputPositions" : 0, "rawInputDataSize" : "181576997B", "inputDataSize" : "2327849012B", "inputPositions" : 5198526, "sumSquaredInputPositions" : 5.188204009348E12, "getOutputCalls" : 5235, "getOutputWall" : "1.04m", "getOutputCpu" : "26.02s", "outputDataSize" : "2375477893B", "outputPositions" : 5198526, "physicalWrittenDataSize" : "0B", "blockedWall" : "0.00ns", "finishCalls" : 0, "finishWall" : "0.00ns", "finishCpu" : "0.00ns", "userMemoryReservation" : "0B", "revocableMemoryReservation" : "0B", "systemMemoryReservation" : "18937313B", "peakUserMemoryReservation" : "0B", "peakSystemMemoryReservation" : "38629863B", "peakRevocableMemoryReservation" : "0B", "peakTotalMemoryReservation" : "38629863B", "spilledDataSize" : "0B" }, { "stageId" : 2, "pipelineId" : 0, "operatorId" : 1, "planNodeId" : "2", "operatorType" : "TaskOutputOperator", "totalDrivers" : 6, "addInputCalls" : 5214, "addInputWall" : "1.16m", "addInputCpu" : "28.98s", "physicalInputDataSize" : "0B", "physicalInputPositions" : 0, "internalNetworkInputDataSize" : "0B", "internalNetworkInputPositions" : 0, "rawInputDataSize" : "0B", "inputDataSize" : "2374995020B", "inputPositions" : 5197502, "sumSquaredInputPositions" : 5.18595134042E12, "getOutputCalls" : 0, "getOutputWall" : "0.00ns", "getOutputCpu" : "0.00ns", "outputDataSize" : "2374995020B", "outputPositions" : 5197502, "physicalWrittenDataSize" : "0B", "blockedWall" : "22.17s", "finishCalls" : 5, "finishWall" : "74.68us", "finishCpu" : "98.00us", "userMemoryReservation" : "0B", "revocableMemoryReservation" : "0B", "systemMemoryReservation" : "0B", "peakUserMemoryReservation" : "0B", "peakSystemMemoryReservation" : "0B", "peakRevocableMemoryReservation" : "0B", "peakTotalMemoryReservation" : "0B", "spilledDataSize" : "0B" } ], "drivers" : [ ] } ] }, "needsPlan" : false } ], "subStages" : [ ], "tables" : { "0" : { "tableName" : "hive.default.testorc", "predicate" : { "columnDomains" : [ ] } } }, "failureCause" : { "type" : "io.prestosql.ExceededMemoryLimitException", "message" : "Query exceeded per-node total memory limit of 50GB [Allocated: 50.01GB, Delta: 429.71kB, Top Consumers: {OrderByOperator=49.99GB, ScanFilterAndProjectOperator=18.06MB}]", "suppressed" : [ ], "stack" : [ "io.prestosql.ExceededMemoryLimitException.exceededLocalTotalMemoryLimit(ExceededMemoryLimitException.java:46)", "io.prestosql.memory.QueryContext.enforceTotalMemoryLimit(QueryContext.java:342)", "io.prestosql.memory.QueryContext.updateSystemMemory(QueryContext.java:175)", "io.prestosql.memory.QueryContext$QueryMemoryReservationHandler.reserveMemory(QueryContext.java:313)", "io.prestosql.memory.context.RootAggregatedMemoryContext.updateBytes(RootAggregatedMemoryContext.java:37)", "io.prestosql.memory.context.ChildAggregatedMemoryContext.updateBytes(ChildAggregatedMemoryContext.java:38)", "io.prestosql.memory.context.SimpleLocalMemoryContext.setBytes(SimpleLocalMemoryContext.java:66)", "io.prestosql.execution.buffer.OutputBufferMemoryManager.updateMemoryUsage(OutputBufferMemoryManager.java:86)", "io.prestosql.execution.buffer.ArbitraryOutputBuffer.enqueue(ArbitraryOutputBuffer.java:220)", "io.prestosql.execution.buffer.LazyOutputBuffer.enqueue(LazyOutputBuffer.java:254)", "io.prestosql.operator.TaskOutputOperator.addInput(TaskOutputOperator.java:149)", "io.prestosql.operator.Driver.processInternal(Driver.java:384)", "io.prestosql.operator.Driver.lambda$processFor$8(Driver.java:283)", "io.prestosql.operator.Driver.tryWithLock(Driver.java:675)", "io.prestosql.operator.Driver.processFor(Driver.java:276)", "io.prestosql.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:1075)", "io.prestosql.execution.executor.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:163)", "io.prestosql.execution.executor.TaskExecutor$TaskRunner.run(TaskExecutor.java:484)", "io.prestosql.$gen.Presto_nulltestversion__20200316_143022_2.run(Unknown Source)", "java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)", "java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)", "java.lang.Thread.run(Thread.java:748)" ], "errorCode" : { "code" : 131079, "name" : "EXCEEDED_LOCAL_MEMORY_LIMIT", "type" : "INSUFFICIENT_RESOURCES" } } } ], "tables" : { } } ], "tables" : { } }, "failureInfo" : { "type" : "io.prestosql.ExceededMemoryLimitException", "message" : "Query exceeded per-node total memory limit of 50GB [Allocated: 50.01GB, Delta: 429.71kB, Top Consumers: {OrderByOperator=49.99GB, ScanFilterAndProjectOperator=18.06MB}]", "suppressed" : [ ], "stack" : [ "io.prestosql.ExceededMemoryLimitException.exceededLocalTotalMemoryLimit(ExceededMemoryLimitException.java:46)", "io.prestosql.memory.QueryContext.enforceTotalMemoryLimit(QueryContext.java:342)", "io.prestosql.memory.QueryContext.updateSystemMemory(QueryContext.java:175)", "io.prestosql.memory.QueryContext$QueryMemoryReservationHandler.reserveMemory(QueryContext.java:313)", "io.prestosql.memory.context.RootAggregatedMemoryContext.updateBytes(RootAggregatedMemoryContext.java:37)", "io.prestosql.memory.context.ChildAggregatedMemoryContext.updateBytes(ChildAggregatedMemoryContext.java:38)", "io.prestosql.memory.context.SimpleLocalMemoryContext.setBytes(SimpleLocalMemoryContext.java:66)", "io.prestosql.execution.buffer.OutputBufferMemoryManager.updateMemoryUsage(OutputBufferMemoryManager.java:86)", "io.prestosql.execution.buffer.ArbitraryOutputBuffer.enqueue(ArbitraryOutputBuffer.java:220)", "io.prestosql.execution.buffer.LazyOutputBuffer.enqueue(LazyOutputBuffer.java:254)", "io.prestosql.operator.TaskOutputOperator.addInput(TaskOutputOperator.java:149)", "io.prestosql.operator.Driver.processInternal(Driver.java:384)", "io.prestosql.operator.Driver.lambda$processFor$8(Driver.java:283)", "io.prestosql.operator.Driver.tryWithLock(Driver.java:675)", "io.prestosql.operator.Driver.processFor(Driver.java:276)", "io.prestosql.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:1075)", "io.prestosql.execution.executor.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:163)", "io.prestosql.execution.executor.TaskExecutor$TaskRunner.run(TaskExecutor.java:484)", "io.prestosql.$gen.Presto_nulltestversion__20200316_143022_2.run(Unknown Source)", "java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)", "java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)", "java.lang.Thread.run(Thread.java:748)" ], "errorCode" : { "code" : 131079, "name" : "EXCEEDED_LOCAL_MEMORY_LIMIT", "type" : "INSUFFICIENT_RESOURCES" } }, "errorCode" : { "code" : 131079, "name" : "EXCEEDED_LOCAL_MEMORY_LIMIT", "type" : "INSUFFICIENT_RESOURCES" }, "warnings" : [ ], "inputs" : [ { "catalogName" : "hive", "schema" : "default", "table" : "testorc", "columns" : [ { "name" : "total_tool_report", "type" : "double" }, { "name" : "email_cost", "type" : "decimal(2,1)" }, { "name" : "downloads", "type" : "double" }, { "name" : "mps_flag", "type" : "decimal(2,1)" }, { "name" : "sfmc_activity_type", "type" : "varchar" }, { "name" : "territory_code_2", "type" : "varchar" }, { "name" : "channel_name", "type" : "varchar" }, { "name" : "gl_lob", "type" : "varchar" }, { "name" : "day", "type" : "date" }, { "name" : "target_date_compass_tool_report", "type" : "double" }, { "name" : "is_hard_bounce", "type" : "double" }, { "name" : "video_views", "type" : "double" }, { "name" : "region", "type" : "varchar" }, { "name" : "subs_frequency", "type" : "varchar(13)" }, { "name" : "is_open", "type" : "double" }, { "name" : "salesforce_contact_id", "type" : "varchar" }, { "name" : "target_date_compass_tool_analysis", "type" : "double" }, { "name" : "company_id_l1", "type" : "varchar" }, { "name" : "podcast_plays", "type" : "double" }, { "name" : "total_opens", "type" : "double" }, { "name" : "territory_code_1", "type" : "varchar" }, { "name" : "portfolio_insights_tool_report", "type" : "double" }, { "name" : "territory_code_5", "type" : "varchar" }, { "name" : "bond_ladder_tool_analysis", "type" : "double" }, { "name" : "is_link_clicked", "type" : "double" }, { "name" : "investment_comparison_tool_report", "type" : "double" }, { "name" : "url_alias_80", "type" : "varchar" }, { "name" : "tier", "type" : "varchar" }, { "name" : "sales_channel", "type" : "varchar" }, { "name" : "guide_to_markets_interactions", "type" : "double" }, { "name" : "campaign_id", "type" : "varchar" }, { "name" : "is_inferred_open", "type" : "double" }, { "name" : "subregion", "type" : "varchar" }, { "name" : "is_sent", "type" : "double" }, { "name" : "secondary_topics", "type" : "varchar" }, { "name" : "primary_topics", "type" : "varchar" }, { "name" : "territory_code_4", "type" : "varchar" }, { "name" : "bond_ladder_tool_report", "type" : "double" }, { "name" : "market_insights_email_signups", "type" : "double" }, { "name" : "country_full", "type" : "varchar" }, { "name" : "email_engaged_sub", "type" : "decimal(2,1)" }, { "name" : "is_sub", "type" : "double" }, { "name" : "sfmc_activity_sub_type", "type" : "varchar" }, { "name" : "investment_comparison_tool_analysis", "type" : "double" }, { "name" : "sfmc_activity_id", "type" : "varchar" }, { "name" : "registrations", "type" : "double" }, { "name" : "is_delivered", "type" : "double" }, { "name" : "campaign_name", "type" : "varchar" }, { "name" : "is_bounce", "type" : "double" }, { "name" : "is_opt_out", "type" : "double" }, { "name" : "email_id", "type" : "varchar" }, { "name" : "mpc_flag", "type" : "integer" }, { "name" : "territory_code_3", "type" : "varchar" }, { "name" : "total_tool_analysis", "type" : "double" }, { "name" : "url_clicked_personalized", "type" : "varchar" }, { "name" : "total", "type" : "double" }, { "name" : "portfolio_insights_tool_analysis", "type" : "double" }, { "name" : "logins", "type" : "double" }, { "name" : "web_engaged_sub", "type" : "decimal(2,1)" }, { "name" : "pageviews", "type" : "double" }, { "name" : "opportunities", "type" : "bigint" } ] } ], "resourceGroupId" : [ "global" ], "errorType" : "INSUFFICIENT_RESOURCES", "finalQueryInfo" : true }

alexey-fin commented 4 years ago

@tooptoop4 i can also increase the memory so it will finish and add a json of that run, should i?

sopel39 commented 4 years ago

Hi @alexey-fin

Do you have some Presto modifications? The Presto version from stack trace shows:

Presto_null__testversion____20200316_143022_2

sopel39 commented 4 years ago

Could you share EXPLAIN

alexey-fin commented 4 years ago

Output[salesforce_contact_id, email_id, campaign_id, sfmc_activity_id, campaign_name, sfmc_activity_type, sfmc_activity_sub_type, day, is_delivered, is_open, is_inferred_open, total_opens, is_link_clicked, is_hard_bounce, is_bounce, is_opt_out, is_sent, gl_lob, region, subregion, company_id_l1, tier, sales_channel, channel_name, primary_topics, secondary_topics, subs_frequency, url_clicked_personalized, url_alias_80, territory_code_1, territory_code_2, territory_code_3, territory_code_4, territory_code_5, country_full, email_cost, web_engaged_sub, email_engaged_sub, mps_flag, opportunities, is_sub, mpc_flag, downloads, video_views, logins, pageviews, guide_to_markets_interactions, market_insights_email_signups, registrations, podcast_plays, bond_ladder_tool_analysis, portfolio_insights_tool_analysis, investment_comparison_tool_analysis, target_date_compass_tool_analysis, bond_ladder_tool_report, portfolio_insights_tool_report, investment_comparison_tool_report, target_date_compass_tool_report, total_tool_report, total_tool_analysis, total]
│   Layout: [salesforce_contact_id:varchar, email_id:varchar, campaign_id:varchar, sfmc_activity_id:varchar, campaign_name:varchar, sfmc_activity_type:varchar, sfmc_activity_sub_type:varchar, day:date, is_delivered:double, is_open:double, is_inferred_open:double, total_opens:double, is_link_clicked:double, is_hard_bounce:double, is_bounce:double, is_opt_out:double, is_sent:double, gl_lob:varchar, region:varchar, subregion:varchar, company_id_l1:varchar, tier:varchar, sales_channel:varchar, channel_name:varchar, primary_topics:varchar, secondary_topics:varchar, subs_frequency:varchar(13), url_clicked_personalized:varchar, url_alias:varchar, territory_code:varchar, territory_code_0:varchar, territory_code_1:varchar, territory_code_2:varchar, territory_code_3:varchar, country_full:varchar, email_cost:decimal(2,1), web_engaged_sub:decimal(2,1), email_engaged_sub:decimal(2,1), mps_flag:decimal(2,1), opportunities:bigint, is_sub:double, mpc_flag:integer, downloads:double, video_views:double, logins:double, pageviews:double, guide_to_markets_interactions:double, market_insights_email_signups:double, registrations:double, podcast_plays:double, bond_ladder_tool_analysis:double, portfolio_insights_tool_analysis:double, investment_comparison_tool_analysis:double, target_date_compass_tool_analysis:double, bond_ladder_tool_report:double, portfolio_insights_tool_report:double, investment_comparison_tool_report:double, target_date_compass_tool_report:double, total_tool_report:double, total_tool_analysis:double, total:double]
│   Estimates: {rows: 0 (0B), cpu: ?, memory: ?, network: ?}
│   url_alias_80 := url_alias
│   territory_code_1 := territory_code
│   territory_code_2 := territory_code_0
│   territory_code_3 := territory_code_1
│   territory_code_4 := territory_code_2
│   territory_code_5 := territory_code_3
└─ Project[]
   │   Layout: [salesforce_contact_id:varchar, email_id:varchar, campaign_id:varchar, sfmc_activity_id:varchar, campaign_name:varchar, sfmc_activity_type:varchar, sfmc_activity_sub_type:varchar, day:date, is_delivered:double, is_open:double, is_inferred_open:double, total_opens:double, is_link_clicked:double, is_hard_bounce:double, is_bounce:double, is_opt_out:double, is_sent:double, gl_lob:varchar, region:varchar, subregion:varchar, company_id_l1:varchar, tier:varchar, sales_channel:varchar, channel_name:varchar, primary_topics:varchar, secondary_topics:varchar, subs_frequency:varchar(13), url_clicked_personalized:varchar, url_alias:varchar, territory_code:varchar, territory_code_0:varchar, territory_code_1:varchar, territory_code_2:varchar, territory_code_3:varchar, country_full:varchar, email_cost:decimal(2,1), web_engaged_sub:decimal(2,1), email_engaged_sub:decimal(2,1), mps_flag:decimal(2,1), opportunities:bigint, is_sub:double, mpc_flag:integer, downloads:double, video_views:double, logins:double, pageviews:double, guide_to_markets_interactions:double, market_insights_email_signups:double, registrations:double, podcast_plays:double, bond_ladder_tool_analysis:double, portfolio_insights_tool_analysis:double, investment_comparison_tool_analysis:double, target_date_compass_tool_analysis:double, bond_ladder_tool_report:double, portfolio_insights_tool_report:double, investment_comparison_tool_report:double, target_date_compass_tool_report:double, total_tool_report:double, total_tool_analysis:double, total:double]
   │   Estimates: {rows: 0 (0B), cpu: ?, memory: ?, network: ?}
   └─ RemoteMerge[random ASC_NULLS_LAST]
      │   Layout: [random:double, salesforce_contact_id:varchar, email_id:varchar, campaign_id:varchar, sfmc_activity_id:varchar, campaign_name:varchar, sfmc_activity_type:varchar, sfmc_activity_sub_type:varchar, day:date, is_delivered:double, is_open:double, is_inferred_open:double, total_opens:double, is_link_clicked:double, is_hard_bounce:double, is_bounce:double, is_opt_out:double, is_sent:double, gl_lob:varchar, region:varchar, subregion:varchar, company_id_l1:varchar, tier:varchar, sales_channel:varchar, channel_name:varchar, primary_topics:varchar, secondary_topics:varchar, subs_frequency:varchar(13), url_clicked_personalized:varchar, url_alias:varchar, territory_code:varchar, territory_code_0:varchar, territory_code_1:varchar, territory_code_2:varchar, territory_code_3:varchar, country_full:varchar, email_cost:decimal(2,1), web_engaged_sub:decimal(2,1), email_engaged_sub:decimal(2,1), mps_flag:decimal(2,1), opportunities:bigint, is_sub:double, mpc_flag:integer, downloads:double, video_views:double, logins:double, pageviews:double, guide_to_markets_interactions:double, market_insights_email_signups:double, registrations:double, podcast_plays:double, bond_ladder_tool_analysis:double, portfolio_insights_tool_analysis:double, investment_comparison_tool_analysis:double, target_date_compass_tool_analysis:double, bond_ladder_tool_report:double, portfolio_insights_tool_report:double, investment_comparison_tool_report:double, target_date_compass_tool_report:double, total_tool_report:double, total_tool_analysis:double, total:double]
      │   Estimates: {rows: 0 (0B), cpu: ?, memory: ?, network: ?}
      └─ LocalMerge[random ASC_NULLS_LAST]
         │   Layout: [random:double, salesforce_contact_id:varchar, email_id:varchar, campaign_id:varchar, sfmc_activity_id:varchar, campaign_name:varchar, sfmc_activity_type:varchar, sfmc_activity_sub_type:varchar, day:date, is_delivered:double, is_open:double, is_inferred_open:double, total_opens:double, is_link_clicked:double, is_hard_bounce:double, is_bounce:double, is_opt_out:double, is_sent:double, gl_lob:varchar, region:varchar, subregion:varchar, company_id_l1:varchar, tier:varchar, sales_channel:varchar, channel_name:varchar, primary_topics:varchar, secondary_topics:varchar, subs_frequency:varchar(13), url_clicked_personalized:varchar, url_alias:varchar, territory_code:varchar, territory_code_0:varchar, territory_code_1:varchar, territory_code_2:varchar, territory_code_3:varchar, country_full:varchar, email_cost:decimal(2,1), web_engaged_sub:decimal(2,1), email_engaged_sub:decimal(2,1), mps_flag:decimal(2,1), opportunities:bigint, is_sub:double, mpc_flag:integer, downloads:double, video_views:double, logins:double, pageviews:double, guide_to_markets_interactions:double, market_insights_email_signups:double, registrations:double, podcast_plays:double, bond_ladder_tool_analysis:double, portfolio_insights_tool_analysis:double, investment_comparison_tool_analysis:double, target_date_compass_tool_analysis:double, bond_ladder_tool_report:double, portfolio_insights_tool_report:double, investment_comparison_tool_report:double, target_date_compass_tool_report:double, total_tool_report:double, total_tool_analysis:double, total:double]
         │   Estimates: {rows: 0 (0B), cpu: ?, memory: ?, network: ?}
         └─ PartialSort[random ASC_NULLS_LAST]
            │   Layout: [random:double, salesforce_contact_id:varchar, email_id:varchar, campaign_id:varchar, sfmc_activity_id:varchar, campaign_name:varchar, sfmc_activity_type:varchar, sfmc_activity_sub_type:varchar, day:date, is_delivered:double, is_open:double, is_inferred_open:double, total_opens:double, is_link_clicked:double, is_hard_bounce:double, is_bounce:double, is_opt_out:double, is_sent:double, gl_lob:varchar, region:varchar, subregion:varchar, company_id_l1:varchar, tier:varchar, sales_channel:varchar, channel_name:varchar, primary_topics:varchar, secondary_topics:varchar, subs_frequency:varchar(13), url_clicked_personalized:varchar, url_alias:varchar, territory_code:varchar, territory_code_0:varchar, territory_code_1:varchar, territory_code_2:varchar, territory_code_3:varchar, country_full:varchar, email_cost:decimal(2,1), web_engaged_sub:decimal(2,1), email_engaged_sub:decimal(2,1), mps_flag:decimal(2,1), opportunities:bigint, is_sub:double, mpc_flag:integer, downloads:double, video_views:double, logins:double, pageviews:double, guide_to_markets_interactions:double, market_insights_email_signups:double, registrations:double, podcast_plays:double, bond_ladder_tool_analysis:double, portfolio_insights_tool_analysis:double, investment_comparison_tool_analysis:double, target_date_compass_tool_analysis:double, bond_ladder_tool_report:double, portfolio_insights_tool_report:double, investment_comparison_tool_report:double, target_date_compass_tool_report:double, total_tool_report:double, total_tool_analysis:double, total:double]
            │   Estimates: {rows: 0 (0B), cpu: ?, memory: ?, network: ?}
            └─ RemoteExchange[REPARTITION]
               │   Layout: [random:double, salesforce_contact_id:varchar, email_id:varchar, campaign_id:varchar, sfmc_activity_id:varchar, campaign_name:varchar, sfmc_activity_type:varchar, sfmc_activity_sub_type:varchar, day:date, is_delivered:double, is_open:double, is_inferred_open:double, total_opens:double, is_link_clicked:double, is_hard_bounce:double, is_bounce:double, is_opt_out:double, is_sent:double, gl_lob:varchar, region:varchar, subregion:varchar, company_id_l1:varchar, tier:varchar, sales_channel:varchar, channel_name:varchar, primary_topics:varchar, secondary_topics:varchar, subs_frequency:varchar(13), url_clicked_personalized:varchar, url_alias:varchar, territory_code:varchar, territory_code_0:varchar, territory_code_1:varchar, territory_code_2:varchar, territory_code_3:varchar, country_full:varchar, email_cost:decimal(2,1), web_engaged_sub:decimal(2,1), email_engaged_sub:decimal(2,1), mps_flag:decimal(2,1), opportunities:bigint, is_sub:double, mpc_flag:integer, downloads:double, video_views:double, logins:double, pageviews:double, guide_to_markets_interactions:double, market_insights_email_signups:double, registrations:double, podcast_plays:double, bond_ladder_tool_analysis:double, portfolio_insights_tool_analysis:double, investment_comparison_tool_analysis:double, target_date_compass_tool_analysis:double, bond_ladder_tool_report:double, portfolio_insights_tool_report:double, investment_comparison_tool_report:double, target_date_compass_tool_report:double, total_tool_report:double, total_tool_analysis:double, total:double]
               │   Estimates: {rows: 0 (0B), cpu: 0, memory: 0B, network: 0B}
               └─ ScanProject[table = hive:default:testorc]
                      Layout: [random:double, salesforce_contact_id:varchar, email_id:varchar, campaign_id:varchar, sfmc_activity_id:varchar, campaign_name:varchar, sfmc_activity_type:varchar, sfmc_activity_sub_type:varchar, day:date, is_delivered:double, is_open:double, is_inferred_open:double, total_opens:double, is_link_clicked:double, is_hard_bounce:double, is_bounce:double, is_opt_out:double, is_sent:double, gl_lob:varchar, region:varchar, subregion:varchar, company_id_l1:varchar, tier:varchar, sales_channel:varchar, channel_name:varchar, primary_topics:varchar, secondary_topics:varchar, subs_frequency:varchar(13), url_clicked_personalized:varchar, url_alias:varchar, territory_code:varchar, territory_code_0:varchar, territory_code_1:varchar, territory_code_2:varchar, territory_code_3:varchar, country_full:varchar, email_cost:decimal(2,1), web_engaged_sub:decimal(2,1), email_engaged_sub:decimal(2,1), mps_flag:decimal(2,1), opportunities:bigint, is_sub:double, mpc_flag:integer, downloads:double, video_views:double, logins:double, pageviews:double, guide_to_markets_interactions:double, market_insights_email_signups:double, registrations:double, podcast_plays:double, bond_ladder_tool_analysis:double, portfolio_insights_tool_analysis:double, investment_comparison_tool_analysis:double, target_date_compass_tool_analysis:double, bond_ladder_tool_report:double, portfolio_insights_tool_report:double, investment_comparison_tool_report:double, target_date_compass_tool_report:double, total_tool_report:double, total_tool_analysis:double, total:double]
                      Estimates: {rows: 0 (0B), cpu: 0, memory: 0B, network: 0B}/{rows: 0 (0B), cpu: 0, memory: 0B, network: 0B}
                      random := random()
                      total_tool_report := total_tool_report:double:58:REGULAR
                      email_cost := email_cost:decimal(2,1):35:REGULAR
                      gl_lob := gl_lob:string:17:REGULAR
                      is_sub := is_sub:double:40:REGULAR
                      subs_frequency := subs_frequency:varchar(13):26:REGULAR
                      territory_code_3 := territory_code_5:string:33:REGULAR
                      territory_code_1 := territory_code_3:string:31:REGULAR
                      secondary_topics := secondary_topics:string:25:REGULAR
                      territory_code_2 := territory_code_4:string:32:REGULAR
                      is_bounce := is_bounce:double:14:REGULAR
                      territory_code_0 := territory_code_2:string:30:REGULAR
                      primary_topics := primary_topics:string:24:REGULAR
                      is_sent := is_sent:double:16:REGULAR
                      video_views := video_views:double:43:REGULAR
                      total_tool_analysis := total_tool_analysis:double:59:REGULAR
                      day := day:date:7:REGULAR
                      investment_comparison_tool_report := investment_comparison_tool_report:double:56:REGULAR
                      market_insights_email_signups := market_insights_email_signups:double:47:REGULAR
                      campaign_id := campaign_id:string:2:REGULAR
                      territory_code := territory_code_1:string:29:REGULAR
                      registrations := registrations:double:48:REGULAR
                      is_open := is_open:double:9:REGULAR
                      subregion := subregion:string:19:REGULAR
                      sales_channel := sales_channel:string:22:REGULAR
                      is_hard_bounce := is_hard_bounce:double:13:REGULAR
                      podcast_plays := podcast_plays:double:49:REGULAR
                      is_inferred_open := is_inferred_open:double:10:REGULAR
                      salesforce_contact_id := salesforce_contact_id:string:0:REGULAR
                      target_date_compass_tool_analysis := target_date_compass_tool_analysis:double:53:REGULAR
                      email_engaged_sub := email_engaged_sub:decimal(2,1):37:REGULAR
                      region := region:string:18:REGULAR
                      logins := logins:double:44:REGULAR
                      email_id := email_id:string:1:REGULAR
                      is_opt_out := is_opt_out:double:15:REGULAR
                      is_delivered := is_delivered:double:8:REGULAR
                      investment_comparison_tool_analysis := investment_comparison_tool_analysis:double:52:REGULAR
                      portfolio_insights_tool_analysis := portfolio_insights_tool_analysis:double:51:REGULAR
                      country_full := country_full:string:34:REGULAR
                      campaign_name := campaign_name:string:4:REGULAR
                      total := total:double:60:REGULAR
                      pageviews := pageviews:double:45:REGULAR
                      url_alias := url_alias_80:string:28:REGULAR
                      bond_ladder_tool_analysis := bond_ladder_tool_analysis:double:50:REGULAR
                      tier := tier:string:21:REGULAR
                      downloads := downloads:double:42:REGULAR
                      is_link_clicked := is_link_clicked:double:12:REGULAR
                      web_engaged_sub := web_engaged_sub:decimal(2,1):36:REGULAR
                      mps_flag := mps_flag:decimal(2,1):38:REGULAR
                      url_clicked_personalized := url_clicked_personalized:string:27:REGULAR
                      sfmc_activity_sub_type := sfmc_activity_sub_type:string:6:REGULAR
                      mpc_flag := mpc_flag:int:41:REGULAR
                      portfolio_insights_tool_report := portfolio_insights_tool_report:double:55:REGULAR
                      channel_name := channel_name:string:23:REGULAR
                      guide_to_markets_interactions := guide_to_markets_interactions:double:46:REGULAR
                      sfmc_activity_type := sfmc_activity_type:string:5:REGULAR
                      opportunities := opportunities:bigint:39:REGULAR
                      company_id_l1 := company_id_l1:string:20:REGULAR
                      total_opens := total_opens:double:11:REGULAR
                      sfmc_activity_id := sfmc_activity_id:string:3:REGULAR
                      bond_ladder_tool_report := bond_ladder_tool_report:double:54:REGULAR
                      target_date_compass_tool_report := target_date_compass_tool_report:double:57:REGULAR

alexey-fin commented 4 years ago

No modifications, just reproducing it locally from master for simplicity

sopel39 commented 4 years ago

Could you try with pages-index.eager-compaction-enabled=true?

alexey-fin commented 4 years ago

@sopel39 yep, running with on, no effect unfortunately

sopel39 commented 4 years ago

@alexey-fin OrderBy operator is pretty trivial in memory accounting.

Are you running the query locally?
Can you reproduce with TPCH/TPC-DS schema?

alexey-fin commented 4 years ago

1) yes, reproduced locally following failures in production 2) select from ( select from tpch.sf100000.part limit 200000000 ) order by random() works ok, peak memory 30gb and similar heap consumption

alexey-fin commented 4 years ago

@sopel39

trinodb / trino

OrderByOperator extremely overestimates the memory usage #3036