Gresliebear
commented
3 years ago Open Gresliebear opened 3 years ago
Gresliebear
commented
3 years ago Gresliebear
commented
3 years ago
/job:localhost/replica:0/task:0/device:CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[]
ResourceAccumulatorTakeGradient: CPU
ResourceConditionalAccumulator: CPU
ResourceAccumulatorApplyGradient: CPU
Colocation members, user-requested devices, and framework assigned devices, if any:
optimization/grad_accumulator/conditional_accumulator_134 (ResourceConditionalAccumulator) /device:GPU:0
optimization/apply_gradient/ResourceAccumulatorApplyGradient_134 (ResourceAccumulatorApplyGradient) /device:GPU:0
optimization/take_gradients/ResourceAccumulatorTakeGradient_134 (ResourceAccumulatorTakeGradient) /device:GPU:0
2021-03-14 05:07:17.334989: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
/job:localhost/replica:0/task:0/device:CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[]
ResourceAccumulatorTakeGradient: CPU
ResourceConditionalAccumulator: CPU
ResourceAccumulatorApplyGradient: CPU
Colocation members, user-requested devices, and framework assigned devices, if any:
optimization/grad_accumulator/conditional_accumulator_135 (ResourceConditionalAccumulator) /device:GPU:0
optimization/apply_gradient/ResourceAccumulatorApplyGradient_135 (ResourceAccumulatorApplyGradient) /device:GPU:0
optimization/take_gradients/ResourceAccumulatorTakeGradient_135 (ResourceAccumulatorTakeGradient) /device:GPU:0
2021-03-14 05:07:17.335084: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
/job:localhost/replica:0/task:0/device:CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[]
ResourceAccumulatorTakeGradient: CPU
ResourceConditionalAccumulator: CPU
ResourceAccumulatorApplyGradient: CPU
Colocation members, user-requested devices, and framework assigned devices, if any:
optimization/grad_accumulator/conditional_accumulator_136 (ResourceConditionalAccumulator) /device:GPU:0
optimization/apply_gradient/ResourceAccumulatorApplyGradient_136 (ResourceAccumulatorApplyGradient) /device:GPU:0
optimization/take_gradients/ResourceAccumulatorTakeGradient_136 (ResourceAccumulatorTakeGradient) /device:GPU:0
2021-03-14 05:07:17.335178: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
/job:localhost/replica:0/task:0/device:CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[]
ResourceAccumulatorTakeGradient: CPU
ResourceConditionalAccumulator: CPU
ResourceAccumulatorApplyGradient: CPU
Colocation members, user-requested devices, and framework assigned devices, if any:
optimization/grad_accumulator/conditional_accumulator_137 (ResourceConditionalAccumulator) /device:GPU:0
optimization/apply_gradient/ResourceAccumulatorApplyGradient_137 (ResourceAccumulatorApplyGradient) /device:GPU:0
optimization/take_gradients/ResourceAccumulatorTakeGradient_137 (ResourceAccumulatorTakeGradient) /device:GPU:0
2021-03-14 05:07:17.335292: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
/job:localhost/replica:0/task:0/device:CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[]
ResourceAccumulatorTakeGradient: CPU
ResourceConditionalAccumulator: CPU
ResourceAccumulatorApplyGradient: CPU
Colocation members, user-requested devices, and framework assigned devices, if any:
optimization/grad_accumulator/conditional_accumulator_138 (ResourceConditionalAccumulator) /device:GPU:0
optimization/apply_gradient/ResourceAccumulatorApplyGradient_138 (ResourceAccumulatorApplyGradient) /device:GPU:0
optimization/take_gradients/ResourceAccumulatorTakeGradient_138 (ResourceAccumulatorTakeGradient) /device:GPU:0
2021-03-14 05:07:17.335390: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
/job:localhost/replica:0/task:0/device:CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[]
ResourceAccumulatorTakeGradient: CPU
ResourceConditionalAccumulator: CPU
ResourceAccumulatorApplyGradient: CPU
Colocation members, user-requested devices, and framework assigned devices, if any:
optimization/grad_accumulator/conditional_accumulator_139 (ResourceConditionalAccumulator) /device:GPU:0
optimization/apply_gradient/ResourceAccumulatorApplyGradient_139 (ResourceAccumulatorApplyGradient) /device:GPU:0
optimization/take_gradients/ResourceAccumulatorTakeGradient_139 (ResourceAccumulatorTakeGradient) /device:GPU:0
2021-03-14 05:07:17.335484: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
/job:localhost/replica:0/task:0/device:CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[]
ResourceAccumulatorTakeGradient: CPU
ResourceConditionalAccumulator: CPU
ResourceAccumulatorApplyGradient: CPU
Colocation members, user-requested devices, and framework assigned devices, if any:
optimization/grad_accumulator/conditional_accumulator_140 (ResourceConditionalAccumulator) /device:GPU:0
optimization/apply_gradient/ResourceAccumulatorApplyGradient_140 (ResourceAccumulatorApplyGradient) /device:GPU:0
optimization/take_gradients/ResourceAccumulatorTakeGradient_140 (ResourceAccumulatorTakeGradient) /device:GPU:0
2021-03-14 05:07:17.335579: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
/job:localhost/replica:0/task:0/device:CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[]
ResourceAccumulatorTakeGradient: CPU
ResourceConditionalAccumulator: CPU
ResourceAccumulatorApplyGradient: CPU
Colocation members, user-requested devices, and framework assigned devices, if any:
optimization/grad_accumulator/conditional_accumulator_141 (ResourceConditionalAccumulator) /device:GPU:0
optimization/apply_gradient/ResourceAccumulatorApplyGradient_141 (ResourceAccumulatorApplyGradient) /device:GPU:0
optimization/take_gradients/ResourceAccumulatorTakeGradient_141 (ResourceAccumulatorTakeGradient) /device:GPU:0
2021-03-14 05:07:17.335675: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
/job:localhost/replica:0/task:0/device:CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[]
ResourceAccumulatorTakeGradient: CPU
ResourceConditionalAccumulator: CPU
ResourceAccumulatorApplyGradient: CPU
Colocation members, user-requested devices, and framework assigned devices, if any:
optimization/grad_accumulator/conditional_accumulator_142 (ResourceConditionalAccumulator) /device:GPU:0
optimization/apply_gradient/ResourceAccumulatorApplyGradient_142 (ResourceAccumulatorApplyGradient) /device:GPU:0
optimization/take_gradients/ResourceAccumulatorTakeGradient_142 (ResourceAccumulatorTakeGradient) /device:GPU:0
2021-03-14 05:07:17.335768: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
/job:localhost/replica:0/task:0/device:CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[]
ResourceAccumulatorTakeGradient: CPU
ResourceConditionalAccumulator: CPU
ResourceAccumulatorApplyGradient: CPU
Colocation members, user-requested devices, and framework assigned devices, if any:
optimization/grad_accumulator/conditional_accumulator_143 (ResourceConditionalAccumulator) /device:GPU:0
optimization/apply_gradient/ResourceAccumulatorApplyGradient_143 (ResourceAccumulatorApplyGradient) /device:GPU:0
optimization/take_gradients/ResourceAccumulatorTakeGradient_143 (ResourceAccumulatorTakeGradient) /device:GPU:0
2021-03-14 05:07:17.335860: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
/job:localhost/replica:0/task:0/device:CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[]
ResourceAccumulatorTakeGradient: CPU
ResourceConditionalAccumulator: CPU
ResourceAccumulatorApplyGradient: CPU
Colocation members, user-requested devices, and framework assigned devices, if any:
optimization/grad_accumulator/conditional_accumulator_144 (ResourceConditionalAccumulator) /device:GPU:0
optimization/apply_gradient/ResourceAccumulatorApplyGradient_144 (ResourceAccumulatorApplyGradient) /device:GPU:0
optimization/take_gradients/ResourceAccumulatorTakeGradient_144 (ResourceAccumulatorTakeGradient) /device:GPU:0
2021-03-14 05:07:17.335952: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
/job:localhost/replica:0/task:0/device:CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[]
ResourceAccumulatorTakeGradient: CPU
ResourceConditionalAccumulator: CPU
ResourceAccumulatorApplyGradient: CPU
Colocation members, user-requested devices, and framework assigned devices, if any:
optimization/grad_accumulator/conditional_accumulator_145 (ResourceConditionalAccumulator) /device:GPU:0
optimization/apply_gradient/ResourceAccumulatorApplyGradient_145 (ResourceAccumulatorApplyGradient) /device:GPU:0
optimization/take_gradients/ResourceAccumulatorTakeGradient_145 (ResourceAccumulatorTakeGradient) /device:GPU:0
2021-03-14 05:07:17.336046: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
/job:localhost/replica:0/task:0/device:CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[]
ResourceAccumulatorTakeGradient: CPU
ResourceConditionalAccumulator: CPU
ResourceAccumulatorApplyGradient: CPU
Colocation members, user-requested devices, and framework assigned devices, if any:
optimization/grad_accumulator/conditional_accumulator_146 (ResourceConditionalAccumulator) /device:GPU:0
optimization/apply_gradient/ResourceAccumulatorApplyGradient_146 (ResourceAccumulatorApplyGradient) /device:GPU:0
optimization/take_gradients/ResourceAccumulatorTakeGradient_146 (ResourceAccumulatorTakeGradient) /device:GPU:0
2021-03-14 05:07:17.336139: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
/job:localhost/replica:0/task:0/device:CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[]
ResourceAccumulatorTakeGradient: CPU
ResourceConditionalAccumulator: CPU
ResourceAccumulatorApplyGradient: CPU
Colocation members, user-requested devices, and framework assigned devices, if any:
optimization/grad_accumulator/conditional_accumulator_147 (ResourceConditionalAccumulator) /device:GPU:0
optimization/apply_gradient/ResourceAccumulatorApplyGradient_147 (ResourceAccumulatorApplyGradient) /device:GPU:0
optimization/take_gradients/ResourceAccumulatorTakeGradient_147 (ResourceAccumulatorTakeGradient) /device:GPU:0
2021-03-14 05:07:17.336232: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
/job:localhost/replica:0/task:0/device:CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[]
ResourceAccumulatorTakeGradient: CPU
ResourceConditionalAccumulator: CPU
ResourceAccumulatorApplyGradient: CPU
Colocation members, user-requested devices, and framework assigned devices, if any:
optimization/grad_accumulator/conditional_accumulator_148 (ResourceConditionalAccumulator) /device:GPU:0
optimization/apply_gradient/ResourceAccumulatorApplyGradient_148 (ResourceAccumulatorApplyGradient) /device:GPU:0
optimization/take_gradients/ResourceAccumulatorTakeGradient_148 (ResourceAccumulatorTakeGradient) /device:GPU:0
2021-03-14 05:07:17.336326: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
/job:localhost/replica:0/task:0/device:CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[]
ResourceAccumulatorTakeGradient: CPU
ResourceConditionalAccumulator: CPU
ResourceAccumulatorApplyGradient: CPU
Colocation members, user-requested devices, and framework assigned devices, if any:
optimization/grad_accumulator/conditional_accumulator_149 (ResourceConditionalAccumulator) /device:GPU:0
optimization/apply_gradient/ResourceAccumulatorApplyGradient_149 (ResourceAccumulatorApplyGradient) /device:GPU:0
optimization/take_gradients/ResourceAccumulatorTakeGradient_149 (ResourceAccumulatorTakeGradient) /device:GPU:0
2021-03-14 05:07:17.336419: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
/job:localhost/replica:0/task:0/device:CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[]
ResourceAccumulatorTakeGradient: CPU
ResourceConditionalAccumulator: CPU
ResourceAccumulatorApplyGradient: CPU
Colocation members, user-requested devices, and framework assigned devices, if any:
optimization/grad_accumulator/conditional_accumulator_150 (ResourceConditionalAccumulator) /device:GPU:0
optimization/apply_gradient/ResourceAccumulatorApplyGradient_150 (ResourceAccumulatorApplyGradient) /device:GPU:0
optimization/take_gradients/ResourceAccumulatorTakeGradient_150 (ResourceAccumulatorTakeGradient) /device:GPU:0
2021-03-14 05:07:17.336514: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
/job:localhost/replica:0/task:0/device:CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[]
ResourceAccumulatorTakeGradient: CPU
ResourceConditionalAccumulator: CPU
ResourceAccumulatorApplyGradient: CPU
Colocation members, user-requested devices, and framework assigned devices, if any:
optimization/grad_accumulator/conditional_accumulator_151 (ResourceConditionalAccumulator) /device:GPU:0
optimization/apply_gradient/ResourceAccumulatorApplyGradient_151 (ResourceAccumulatorApplyGradient) /device:GPU:0
optimization/take_gradients/ResourceAccumulatorTakeGradient_151 (ResourceAccumulatorTakeGradient) /device:GPU:0
2021-03-14 05:07:17.336607: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
/job:localhost/replica:0/task:0/device:CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[]
ResourceAccumulatorTakeGradient: CPU
ResourceConditionalAccumulator: CPU
ResourceAccumulatorApplyGradient: CPU
Colocation members, user-requested devices, and framework assigned devices, if any:
optimization/grad_accumulator/conditional_accumulator_152 (ResourceConditionalAccumulator) /device:GPU:0
optimization/apply_gradient/ResourceAccumulatorApplyGradient_152 (ResourceAccumulatorApplyGradient) /device:GPU:0
optimization/take_gradients/ResourceAccumulatorTakeGradient_152 (ResourceAccumulatorTakeGradient) /device:GPU:0
2021-03-14 05:07:17.336728: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
/job:localhost/replica:0/task:0/device:CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[]
ResourceAccumulatorTakeGradient: CPU
ResourceConditionalAccumulator: CPU
ResourceAccumulatorApplyGradient: CPU
Colocation members, user-requested devices, and framework assigned devices, if any:
optimization/grad_accumulator/conditional_accumulator_153 (ResourceConditionalAccumulator) /device:GPU:0
optimization/apply_gradient/ResourceAccumulatorApplyGradient_153 (ResourceAccumulatorApplyGradient) /device:GPU:0
optimization/take_gradients/ResourceAccumulatorTakeGradient_153 (ResourceAccumulatorTakeGradient) /device:GPU:0
2021-03-14 05:07:17.336824: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
/job:localhost/replica:0/task:0/device:CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[]
ResourceAccumulatorTakeGradient: CPU
ResourceConditionalAccumulator: CPU
ResourceAccumulatorApplyGradient: CPU
Colocation members, user-requested devices, and framework assigned devices, if any:
optimization/grad_accumulator/conditional_accumulator_154 (ResourceConditionalAccumulator) /device:GPU:0
optimization/apply_gradient/ResourceAccumulatorApplyGradient_154 (ResourceAccumulatorApplyGradient) /device:GPU:0
optimization/take_gradients/ResourceAccumulatorTakeGradient_154 (ResourceAccumulatorTakeGradient) /device:GPU:0
2021-03-14 05:07:17.336915: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
/job:localhost/replica:0/task:0/device:CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[]
ResourceAccumulatorTakeGradient: CPU
ResourceConditionalAccumulator: CPU
ResourceAccumulatorApplyGradient: CPU
Colocation members, user-requested devices, and framework assigned devices, if any:
optimization/grad_accumulator/conditional_accumulator_155 (ResourceConditionalAccumulator) /device:GPU:0
optimization/apply_gradient/ResourceAccumulatorApplyGradient_155 (ResourceAccumulatorApplyGradient) /device:GPU:0
optimization/take_gradients/ResourceAccumulatorTakeGradient_155 (ResourceAccumulatorTakeGradient) /device:GPU:0
2021-03-14 05:07:17.337007: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
/job:localhost/replica:0/task:0/device:CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[]
ResourceAccumulatorTakeGradient: CPU
ResourceConditionalAccumulator: CPU
ResourceAccumulatorApplyGradient: CPU
Colocation members, user-requested devices, and framework assigned devices, if any:
optimization/grad_accumulator/conditional_accumulator_156 (ResourceConditionalAccumulator) /device:GPU:0
optimization/apply_gradient/ResourceAccumulatorApplyGradient_156 (ResourceAccumulatorApplyGradient) /device:GPU:0
optimization/take_gradients/ResourceAccumulatorTakeGradient_156 (ResourceAccumulatorTakeGradient) /device:GPU:0
2021-03-14 05:07:17.337100: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
/job:localhost/replica:0/task:0/device:CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[]
ResourceAccumulatorTakeGradient: CPU
ResourceConditionalAccumulator: CPU
ResourceAccumulatorApplyGradient: CPU
Colocation members, user-requested devices, and framework assigned devices, if any:
optimization/grad_accumulator/conditional_accumulator_157 (ResourceConditionalAccumulator) /device:GPU:0
optimization/apply_gradient/ResourceAccumulatorApplyGradient_157 (ResourceAccumulatorApplyGradient) /device:GPU:0
optimization/take_gradients/ResourceAccumulatorTakeGradient_157 (ResourceAccumulatorTakeGradient) /device:GPU:0
2021-03-14 05:07:17.337193: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
/job:localhost/replica:0/task:0/device:CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[]
ResourceAccumulatorTakeGradient: CPU
ResourceConditionalAccumulator: CPU
ResourceAccumulatorApplyGradient: CPU
Colocation members, user-requested devices, and framework assigned devices, if any:
optimization/grad_accumulator/conditional_accumulator_158 (ResourceConditionalAccumulator) /device:GPU:0
optimization/apply_gradient/ResourceAccumulatorApplyGradient_158 (ResourceAccumulatorApplyGradient) /device:GPU:0
optimization/take_gradients/ResourceAccumulatorTakeGradient_158 (ResourceAccumulatorTakeGradient) /device:GPU:0
2021-03-14 05:07:17.337287: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
/job:localhost/replica:0/task:0/device:CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[]
ResourceAccumulatorTakeGradient: CPU
ResourceConditionalAccumulator: CPU
ResourceAccumulatorApplyGradient: CPU
Colocation members, user-requested devices, and framework assigned devices, if any:
optimization/grad_accumulator/conditional_accumulator_159 (ResourceConditionalAccumulator) /device:GPU:0
optimization/apply_gradient/ResourceAccumulatorApplyGradient_159 (ResourceAccumulatorApplyGradient) /device:GPU:0
optimization/take_gradients/ResourceAccumulatorTakeGradient_159 (ResourceAccumulatorTakeGradient) /device:GPU:0
2021-03-14 05:07:17.337379: W tensorflow/core/common_runtime/colocation_graph.cc:983] Failed to place the graph without changing the devices of some resources. Some of the operations (that had to be colocated with resource generating operations) are not supported on the resources' devices. Current candidate devices are [
/job:localhost/replica:0/task:0/device:CPU:0].
See below for details of this colocation group:
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[]
ResourceAccumulatorTakeGradient: CPU
ResourceConditionalAccumulator: CPU
ResourceAccumulatorApplyGradient: CPU
Colocation members, user-requested devices, and framework assigned devices, if any:
optimization/grad_accumulator/conditional_accumulator_160 (ResourceConditionalAccumulator) /device:GPU:0
optimization/apply_gradient/ResourceAccumulatorApplyGradient_160 (ResourceAccumulatorApplyGradient) /device:GPU:0
optimization/take_gradients/ResourceAccumulatorTakeGradient_160 (ResourceAccumulatorTakeGradient) /device:GPU:0
Initializing from pre-trained imagenet model...
INFO:tensorflow:Restoring parameters from /media/sdb1/liver_open/liverseg-2017-nipsws/train_files/resnet_v1_50.ckpt
Weights initialized
Start training
2021-03-14 05:07:23.852206 Iter 2: Training Loss = nan
2021-03-14 05:07:23.852246 Iter 2: Validation Loss = nan
2021-03-14 05:07:23.852254 Iter 2: Training Accuracy = 0.5192
2021-03-14 05:07:23.852260 Iter 2: Validation Accuracy = 0.5214
Traceback (most recent call last):
File "main.py", line 171, in <module>
liver_lesion.train()
File "main.py", line 150, in train
step()
File "main.py", line 83, in det_lesion_train
self.det_save_step, self.display_step, self.det_learning_rate)
File "/media/sdb1/liver_open/liverseg-2017-nipsws/det_lesion_train.py", line 56, in det_lesion_train
resume_training = False) # Make true to restore training from a previously saved checkpoint.
File "/media/sdb1/liver_open/liverseg-2017-nipsws/det_lesion.py", line 380, in train
feed_dict = {input_image: val_image, input_label: label_val, is_training: False}) #!!! It was false in the original code. We set it to true. Need to look into it more.
File "/media/sdb1/liver_open/test_env/local/lib/python2.7/site-packages/tensorflow_core/python/client/session.py", line 956, in run
run_metadata_ptr)
File "/media/sdb1/liver_open/test_env/local/lib/python2.7/site-packages/tensorflow_core/python/client/session.py", line 1180, in _run
feed_dict_tensor, options, run_metadata)
File "/media/sdb1/liver_open/test_env/local/lib/python2.7/site-packages/tensorflow_core/python/client/session.py", line 1359, in _do_run
run_metadata)
File "/media/sdb1/liver_open/test_env/local/lib/python2.7/site-packages/tensorflow_core/python/client/session.py", line 1384, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Nan in summary histogram for: losses/losses/output
[[node losses/losses/output (defined at /media/sdb1/liver_open/test_env/local/lib/python2.7/site-packages/tensorflow_core/python/framework/ops.py:1748) ]]
Original stack trace for u'losses/losses/output':
File "main.py", line 171, in <module>
liver_lesion.train()
File "main.py", line 150, in train
step()
File "main.py", line 83, in det_lesion_train
self.det_save_step, self.display_step, self.det_learning_rate)
File "/media/sdb1/liver_open/liverseg-2017-nipsws/det_lesion_train.py", line 56, in det_lesion_train
resume_training = False) # Make true to restore training from a previously saved checkpoint.
File "/media/sdb1/liver_open/liverseg-2017-nipsws/det_lesion.py", line 273, in train
tf.summary.histogram('losses/output', output)
File "/media/sdb1/liver_open/test_env/local/lib/python2.7/site-packages/tensorflow_core/python/summary/summary.py", line 179, in histogram
tag=tag, values=values, name=scope)
File "/media/sdb1/liver_open/test_env/local/lib/python2.7/site-packages/tensorflow_core/python/ops/gen_logging_ops.py", line 329, in histogram_summary
"HistogramSummary", tag=tag, values=values, name=name)
File "/media/sdb1/liver_open/test_env/local/lib/python2.7/site-packages/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper
op_def=op_def)
File "/media/sdb1/liver_open/test_env/local/lib/python2.7/site-packages/tensorflow_core/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/media/sdb1/liver_open/test_env/local/lib/python2.7/site-packages/tensorflow_core/python/framework/ops.py", line 3357, in create_op
attrs, op_def, compute_device)
File "/media/sdb1/liver_open/test_env/local/lib/python2.7/site-packages/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal
op_def=op_def)
File "/media/sdb1/liver_open/test_env/local/lib/python2.7/site-packages/tensorflow_core/python/framework/ops.py", line 1748, in __init__
self._traceback = tf_stack.extract_stack()```