昇腾和寒武纪相关代码退场 Tracking Issue

GreatV commented 1 year ago

一、项目说明用于跟踪昇腾和寒武纪相关代码退场并记录该计划下各个子任务的计划和进度。

二、主要工作 (18/18)

按 merge 的时间顺序，排名不分先后： @KimBioInfoStudio (5) @mrcangye (2) @jjyaoao (3) @Young-Flash (1) @Liyulingyue (3) @engineer1109 (1) @GreatV (2)

任务编号	任务内容	认领人	PR链接
1	删除 `paddle/fluid/platform/device/mlu` 目录✅(2023/3/31)	@mrcangye	#52382
2	删除 `paddle/fluid/operators/_mlu.` 文件✅(2023/4/3)	@Young-Flash	#52435
3	清理 `WITH_MLU` 相关代码✅(2023/4/13)	@GreatV @jjyaoao	#52546✅ #52825✅
4	清理 `is_compiled_with_mlu` 相关代码✅(2023/4/3)	@KimBioInfoStudio	#52378
5	删除 `paddle/fluid/operators/_npu.`、清理 `WITH_ASCEND_INT64` 相关代码✅(2023/4/11)	@jjyaoao @mrcangye	#52673 #52674 #52675 #52676 #52677 #52678 #52679 #53342
6	删除 `/paddle/fluid/platform/device/npu` 目录✅(2023/4/1)	@jjyaoao	#52384
7	清理 `WITH_ASCEND_CL PADDLE_WITH_ASCEND_CL WITH_ASCEND_CXX11` 相关代码✅(2023/4/13)	@engineer1109 @Liyulingyue @jjyaoao	#52448✅ #52535✅ #52612✅ #52825✅
8	清理 `is_compiled_with_npu` 相关代码✅(2023/4/6)	@KimBioInfoStudio	#52385
9	清理 `PADDLE_WITH_ASCEND` 相关代码✅(2023/4/13)	@mrcangye @jjyaoao	#52403✅ #52830✅
10	清理 `WITH_ASCEND` 相关代码✅(2023/4/10)	@Liyulingyue	#52669
11	清理`paddle_build.sh` 脚本中所有的`WITH_MLU` `WITH_ASCEND*`关键字✅(2023/3/31)	@KimBioInfoStudio	#52373
12	清理 `CNCL` 相关代码✅(2023/4/17)	@KimBioInfoStudio	#52811✅ #52434✅ #52939
13	清理 `HCCL` 相关代码✅(2023/4/20)	@Liyulingyue	#52798✅ #52934✅ #52942✅ #53058✅
14	清理 Dockerfile✅(2023/4/4)	@KimBioInfoStudio	#52474
15	清理 `is_mlu_place` `is_npu_place` `is_npu_pinned_place` `SupportNPU` `SupportMLU` 相关代码✅(2023/4/28)	@jjyaoao @Liyulingyue	#52911✅ #53105✅ #53391✅
16	清理 ASCEND 相关代码 ✅(2023/4/24)	@GreatV	#53046✅ #53132✅
17	清理 MLU 相关代码✅(2023/4/24)	@Liyulingyue	#53007✅ #53194✅
18	清理 NPU 相关代码✅(2023/5/18)	@KimBioInfoStudio @Liyulingyue @jjyaoao	#53469✅ #53566✅ #53568✅ #53677✅ #53699✅ #53869✅ #53870✅

需要清理的具体文件可参考此链接。

GreatV commented 1 year ago

1-3. 清理`WITH_MLU`相关代码，删除 `paddle/fluid/platform/device/mlu`目录，删除 `paddle/fluid/operators/*_mlu.*` 文件。具体需清理的文件如下所示：

- `./CMakeLists.txt` - `./test/CMakeLists.txt` - `./cmake/operators.cmake` - `./cmake/third_party.cmake` - `./cmake/configure.cmake` - `./cmake/neuware.cmake` - `./python/CMakeLists.txt` - `./python/paddle/device/__init__.py` - `./paddle/fluid/imperative/CMakeLists.txt` - `./paddle/fluid/imperative/prepared_operator.cc` - `./paddle/fluid/imperative/tracer.cc` - `./paddle/fluid/imperative/gradient_accumulator.cc` - `./paddle/fluid/imperative/amp_auto_cast.cc` - `./paddle/fluid/memory/memcpy.h` - `./paddle/fluid/memory/memcpy.cc` - `./paddle/fluid/memory/allocation/buddy_allocator.cc` - `./paddle/fluid/memory/allocation/buddy_allocator_test.cc` - `./paddle/fluid/memory/allocation/allocator_facade.cc` - `./paddle/fluid/memory/allocation/system_allocator.cc` - `./paddle/fluid/memory/allocation/system_allocator_test.cc` - `./paddle/fluid/memory/allocation/naive_best_fit_allocator.cc` - `./paddle/fluid/memory/allocation/system_allocator.h` - `./paddle/fluid/memory/allocation/naive_best_fit_allocator_test.cc` - `./paddle/fluid/operators/reader/buffered_reader.cc` - `./paddle/fluid/operators/reader/buffered_reader.h` - `./paddle/fluid/operators/where_op_mlu.cc` - `./paddle/fluid/operators/utils.h` - `./paddle/fluid/operators/CMakeLists.txt` - `./paddle/fluid/operators/cast_op.cc` - `./paddle/fluid/operators/reduce_ops/reduce_op_mlu.h` - `./paddle/fluid/operators/mlu/CMakeLists.txt` - `./paddle/fluid/operators/squeeze_op_mlu.cc` - `./paddle/fluid/operators/math/CMakeLists.txt` - `./paddle/fluid/operators/math/concat_and_split.cc` - `./paddle/fluid/operators/expand_v2_op.h` - `./paddle/fluid/operators/unsqueeze_op_mlu.cc` - `./paddle/fluid/operators/elementwise/elementwise_max_op_mlu.cc` - `./paddle/fluid/operators/elementwise/elementwise_mlu.h` - `./paddle/fluid/operators/tile_op_mlu.cc` - `./paddle/fluid/operators/coalesce_tensor_op.cc` - `./paddle/fluid/operators/shape_op_mlu.cc` - `./paddle/fluid/operators/expand_v2_op_mlu.cc` - `./paddle/fluid/operators/detection/CMakeLists.txt` - `./paddle/fluid/operators/softmax_with_cross_entropy_op.cc` - `./paddle/fluid/framework/garbage_collector.cc` - `./paddle/fluid/framework/executor.cc` - `./paddle/fluid/framework/parallel_executor.cc` - `./paddle/fluid/framework/tensor_util.h` - `./paddle/fluid/framework/phi_utils.cc` - `./paddle/fluid/framework/tensor_util.cc` - `./paddle/fluid/framework/operator.cc` - `./paddle/fluid/framework/garbage_collector.h` - `./paddle/fluid/platform/init.cc` - `./paddle/fluid/platform/CMakeLists.txt` - `./paddle/fluid/platform/device_context.cc` - `./paddle/fluid/platform/device_context.h` - `./paddle/fluid/platform/profiler_helper.h` - `./paddle/fluid/platform/init_test.cc` - `./paddle/fluid/platform/device/CMakeLists.txt` - `./paddle/fluid/platform/device/mlu/mlu_info.h` - `./paddle/fluid/platform/device/mlu/CMakeLists.txt` - `./paddle/fluid/platform/device/mlu/device_context.cc` - `./paddle/fluid/platform/device/mlu/device_context.h` - `./paddle/fluid/platform/device/mlu/mlu_resource_pool.cc` - `./paddle/fluid/platform/device/mlu/mlu_stream.h` - `./paddle/fluid/platform/device/mlu/enforce_test.cc` - `./paddle/fluid/platform/device/mlu/mlu_resource_pool.h` - `./paddle/fluid/platform/device/mlu/enforce.h` - `./paddle/fluid/platform/device/mlu/mlu_stream.cc` - `./paddle/fluid/platform/device/device_wrapper.h` - `./paddle/fluid/platform/stream_callback_manager.cc` - `./paddle/fluid/platform/profiler/profiler.cc` - `./paddle/fluid/platform/profiler/mlu/CMakeLists.txt` - `./paddle/fluid/platform/profiler/mlu/cnpapi_data_process.h` - `./paddle/fluid/platform/profiler/mlu/mlu_tracer.h` - `./paddle/fluid/platform/profiler/mlu/mlu_tracer.cc` - `./paddle/fluid/platform/profiler/mlu/cnpapi_data_process.cc` - `./paddle/fluid/platform/profiler/chrometracing_logger.cc` - `./paddle/fluid/platform/place.h` - `./paddle/fluid/pybind/tensor_py.h` - `./paddle/fluid/pybind/tensor.cc` - `./paddle/fluid/pybind/parallel_executor.cc` - `./paddle/fluid/pybind/pybind.cc` - `./paddle/fluid/pybind/place.cc` - `./paddle/phi/kernels/funcs/strided_memcpy.h` - `./paddle/phi/kernels/funcs/activation_functor.h` - `./paddle/phi/core/utils/visit_place.h` - `./paddle/phi/core/flags.cc` - `./paddle/phi/backends/device_memory_aligment.h` - `./paddle/phi/backends/mlu/mlu_info.h` - `./paddle/phi/backends/context_pool.cc`

4. 清理 `is_compiled_with_mlu` 相关代码，具体如下：

- `./tools/get_quick_disable_lt.py` - `./python/paddle/distributed/collective.py` - `./python/paddle/distributed/launch/context/device.py` - `./python/paddle/distributed/fleet/launch.py` - `./python/paddle/distributed/fleet/launch_utils.py` - `./python/paddle/distributed/parallel.py` - `./python/paddle/__init__.py` - `./python/paddle/fluid/dygraph/varbase_patch_methods.py` - `./python/paddle/fluid/tests/unittests/test_dist_base.py` - `./python/paddle/fluid/tests/unittests/eager_op_test.py` - `./python/paddle/fluid/framework.py` - `./python/paddle/fluid/executor.py` - `./python/paddle/static/amp/fp16_lists.py` - `./python/paddle/device/__init__.py` - `./paddle/fluid/pybind/pybind.cc`

5-7. 清理`WITH_ASCEND_CL PADDLE_WITH_ASCEND_CL WITH_ASCEND_CXX11 WITH_ASCEND_INT64`相关代码，删除 `paddle/fluid/operators/*_npu.*`，删除 `/paddle/fluid/platform/device/npu`目录。

- `./tools/coverage/paddle_coverage_new.sh` - `./tools/coverage/paddle_coverage.sh` - `./CMakeLists.txt` - `./test/asp/CMakeLists.txt` - `./cmake/operators.cmake` - `./cmake/inference_lib.cmake` - `./cmake/third_party.cmake` - `./cmake/configure.cmake` - `./cmake/external/warpctc.cmake` - `./cmake/external/protobuf.cmake` - `./cmake/external/threadpool.cmake` - `./cmake/external/gloo.cmake` - `./cmake/external/ascend.cmake` - `./cmake/flags.cmake` - `./python/CMakeLists.txt` - `./python/paddle/fluid/tests/unittests/CMakeLists.txt` - `./python/paddle/fluid/tests/unittests/collective/CMakeLists.txt` - `./python/paddle/fluid/tests/unittests/collective/fleet/CMakeLists.txt` - `./python/paddle/fluid/tests/unittests/distributed_passes/CMakeLists.txt` - `./python/paddle/device/__init__.py` - `./paddle/fluid/imperative/CMakeLists.txt` - `./paddle/fluid/imperative/prepared_operator.cc` - `./paddle/fluid/imperative/tracer.cc` - `./paddle/fluid/imperative/heter_ccl_context.cc` - `./paddle/fluid/imperative/gradient_accumulator.cc` - `./paddle/fluid/imperative/amp_auto_cast.cc` - `./paddle/fluid/imperative/heter_ccl_context.h` - `./paddle/fluid/imperative/hccl_context.h` - `./paddle/fluid/imperative/reducer.h` - `./paddle/fluid/imperative/reducer.cc` - `./paddle/fluid/memory/memcpy.cc` - `./paddle/fluid/memory/allocation/buddy_allocator.cc` - `./paddle/fluid/memory/allocation/buddy_allocator_test.cc` - `./paddle/fluid/memory/allocation/allocator_facade.h` - `./paddle/fluid/memory/allocation/npu_pinned_allocator.h` - `./paddle/fluid/memory/allocation/CMakeLists.txt` - `./paddle/fluid/memory/allocation/allocator_facade.cc` - `./paddle/fluid/memory/allocation/system_allocator.cc` - `./paddle/fluid/memory/allocation/system_allocator_test.cc` - `./paddle/fluid/memory/allocation/npu_pinned_allocator.cc` - `./paddle/fluid/memory/allocation/naive_best_fit_allocator.cc` - `./paddle/fluid/memory/allocation/system_allocator.h` - `./paddle/fluid/memory/allocation/naive_best_fit_allocator_test.cc` - `./paddle/fluid/operators/reader/buffered_reader.cc` - `./paddle/fluid/operators/reader/buffered_reader.h` - `./paddle/fluid/operators/memcpy_op.cc` - `./paddle/fluid/operators/memcpy_h2d_op.cc` - `./paddle/fluid/operators/utils.h` - `./paddle/fluid/operators/scatter_op_npu.cc` - `./paddle/fluid/operators/memcpy_d2h_op.cc` - `./paddle/fluid/operators/CMakeLists.txt` - `./paddle/fluid/operators/copy_cross_scope_test.cc` - `./paddle/fluid/operators/expand_op.h` - `./paddle/fluid/operators/reduce_ops/CMakeLists.txt` - `./paddle/fluid/operators/collective/c_sync_calc_stream_op.h` - `./paddle/fluid/operators/collective/partial_recv_op.cc` - `./paddle/fluid/operators/collective/partial_send_op.cc` - `./paddle/fluid/operators/collective/c_allreduce_max_op_npu_test.cc` - `./paddle/fluid/operators/collective/c_reduce_sum_op_npu_test.cc` - `./paddle/fluid/operators/collective/CMakeLists.txt` - `./paddle/fluid/operators/collective/checknumeric_npu_test.cc` - `./paddle/fluid/operators/collective/c_allgather_op_npu_test.cc` - `./paddle/fluid/operators/collective/c_reduce_op.h` - `./paddle/fluid/operators/collective/gen_hccl_id_op.cc` - `./paddle/fluid/operators/collective/c_broadcast_op_npu.cc` - `./paddle/fluid/operators/collective/send_v2_op_npu_test.cc` - `./paddle/fluid/operators/collective/c_reducescatter_op.cc` - `./paddle/fluid/operators/collective/send_v2_op.cc` - `./paddle/fluid/operators/collective/c_broadcast_op.cc` - `./paddle/fluid/operators/collective/mp_allreduce_sum_op.cc` - `./paddle/fluid/operators/collective/partial_allgather_op_npu.cc` - `./paddle/fluid/operators/collective/c_allgather_op_npu.cc` - `./paddle/fluid/operators/collective/recv_v2_op.cc` - `./paddle/fluid/operators/collective/partial_allgather_op.cc` - `./paddle/fluid/operators/collective/c_comm_init_hccl_op.cc` - `./paddle/fluid/operators/collective/c_reducescatter_op_npu_test.cc` - `./paddle/fluid/operators/collective/c_broadcast_op_npu_test.cc` - `./paddle/fluid/operators/collective/c_gen_hccl_id_op.cc` - `./paddle/fluid/operators/collective/c_allgather_op.cc` - `./paddle/fluid/operators/collective/recv_v2_op_npu.cc` - `./paddle/fluid/operators/collective/c_sync_comm_stream_op.h` - `./paddle/fluid/operators/collective/partial_send_op_npu.cc` - `./paddle/fluid/operators/collective/c_allreduce_op.h` - `./paddle/fluid/operators/collective/c_reducescatter_op_npu.cc` - `./paddle/fluid/operators/collective/gen_hccl_id_op_helper.cc` - `./paddle/fluid/operators/collective/partial_recv_op_npu.cc` - `./paddle/fluid/operators/collective/c_allreduce_sum_op_npu_test.cc` - `./paddle/fluid/operators/collective/recv_v2_op_npu_test.cc` - `./paddle/fluid/operators/collective/c_sync_comm_stream_op_npu_test.cc` - `./paddle/fluid/operators/collective/send_v2_op_npu.cc` - `./paddle/fluid/operators/norm_op.cc` - `./paddle/fluid/operators/run_program_op_npu.cc` - `./paddle/fluid/operators/tensor_formatter.cc` - `./paddle/fluid/operators/math/CMakeLists.txt` - `./paddle/fluid/operators/math/concat_and_split.cc` - `./paddle/fluid/operators/expand_v2_op.h` - `./paddle/fluid/operators/elementwise/CMakeLists.txt` - `./paddle/fluid/operators/amp/check_finite_and_unscale_op.cc` - `./paddle/fluid/operators/amp/CMakeLists.txt` - `./paddle/fluid/operators/unsqueeze_op_npu.cc` - `./paddle/fluid/operators/memcpy_op.h` - `./paddle/fluid/operators/coalesce_tensor_op.cc` - `./paddle/fluid/operators/detection/CMakeLists.txt` - `./paddle/fluid/operators/controlflow/while_op_helper.cc` - `./paddle/fluid/operators/controlflow/conditional_block_op.h` - `./paddle/fluid/operators/softmax_with_cross_entropy_op.cc` - `./paddle/fluid/framework/pipeline_trainer.cc` - `./paddle/fluid/framework/garbage_collector.cc` - `./paddle/fluid/framework/section_worker.cc` - `./paddle/fluid/framework/executor.cc` - `./paddle/fluid/framework/tensor_util_test.cc` - `./paddle/fluid/framework/parallel_executor.cc` - `./paddle/fluid/framework/tensor_util.h` - `./paddle/fluid/framework/details/CMakeLists.txt` - `./paddle/fluid/framework/details/nan_inf_utils_detail.cc` - `./paddle/fluid/framework/details/nan_inf_utils.h` - `./paddle/fluid/framework/type_defs.h` - `./paddle/fluid/framework/phi_utils.cc` - `./paddle/fluid/framework/trainer_factory.cc` - `./paddle/fluid/framework/tensor_util.cc` - `./paddle/fluid/framework/trainer.h` - `./paddle/fluid/framework/var_type_traits.h` - `./paddle/fluid/framework/tensor_test.cc` - `./paddle/fluid/framework/operator.cc` - `./paddle/fluid/framework/fleet/CMakeLists.txt` - `./paddle/fluid/framework/fleet/ascend_wrapper.h` - `./paddle/fluid/framework/fleet/ascend_wrapper.cc` - `./paddle/fluid/framework/new_executor/interpreter/execution_config.cc` - `./paddle/fluid/framework/new_executor/interpreter/interpreter_util.cc` - `./paddle/fluid/framework/new_executor/interpretercore.cc` - `./paddle/fluid/framework/device_worker.h` - `./paddle/fluid/framework/garbage_collector.h` - `./paddle/fluid/framework/device_worker_factory.cc` - `./paddle/fluid/distributed/fleet_executor/test/CMakeLists.txt` - `./paddle/fluid/distributed/fleet_executor/message_bus.cc` - `./paddle/fluid/platform/init.cc` - `./paddle/fluid/platform/CMakeLists.txt` - `./paddle/fluid/platform/device_context.cc` - `./paddle/fluid/platform/device_context.h` - `./paddle/fluid/platform/gen_comm_id_helper.cc` - `./paddle/fluid/platform/gen_comm_id_helper.h` - `./paddle/fluid/platform/device_event_npu.cc` - `./paddle/fluid/platform/device/CMakeLists.txt` - `./paddle/fluid/platform/device/device_wrapper.h` - `./paddle/fluid/platform/device/npu/npu_resource_pool.h` - `./paddle/fluid/platform/device/npu/CMakeLists.txt` - `./paddle/fluid/platform/device/npu/enforce_npu.h` - `./paddle/fluid/platform/device/npu/npu_resource_pool.cc` - `./paddle/fluid/platform/device/npu/hccl_helper.h` - `./paddle/fluid/platform/device/npu/npu_op_runner.h` - `./paddle/fluid/platform/device/npu/npu_info.h` - `./paddle/fluid/platform/device/npu/npu_collective_helper.cc` - `./paddle/fluid/platform/device/npu/npu_stream.h` - `./paddle/fluid/platform/device/npu/ascend_npu_info.h` - `./paddle/fluid/platform/device/npu/dynload/CMakeLists.txt` - `./paddle/fluid/platform/device/npu/dynload/hccl.h` - `./paddle/fluid/platform/device/npu/dynload/hccl.cc` - `./paddle/fluid/platform/stream_callback_manager.cc` - `./paddle/fluid/platform/place.h` - `./paddle/fluid/platform/collective_helper.h` - `./paddle/fluid/platform/dynload/CMakeLists.txt` - `./paddle/fluid/platform/device_event.h` - `./paddle/fluid/inference/analysis/passes/ir_params_sync_among_devices_pass.h` - `./paddle/fluid/inference/analysis/passes/ir_params_sync_among_devices_pass.cc` - `./paddle/fluid/inference/tests/api/CMakeLists.txt` - `./paddle/fluid/inference/api/analysis_predictor.cc` - `./paddle/fluid/inference/api/analysis_config.cc` - `./paddle/fluid/inference/api/details/zero_copy_tensor_test.cc` - `./paddle/fluid/inference/api/details/zero_copy_tensor.cc` - `./paddle/fluid/inference/api/api_impl_tester.cc` - `./paddle/fluid/inference/api/api_impl.cc` - `./paddle/fluid/pybind/CMakeLists.txt` - `./paddle/fluid/pybind/tensor_py.h` - `./paddle/fluid/pybind/ascend_wrapper_py.h` - `./paddle/fluid/pybind/eager_legacy_op_function_generator.cc` - `./paddle/fluid/pybind/tensor.cc` - `./paddle/fluid/pybind/ascend_wrapper_py.cc` - `./paddle/fluid/pybind/parallel_executor.cc` - `./paddle/fluid/pybind/inference_api.cc` - `./paddle/fluid/pybind/imperative.cc` - `./paddle/fluid/pybind/pybind.cc` - `./paddle/fluid/pybind/place.cc` - `./paddle/testing/paddle_gtest_main.cc` - `./paddle/scripts/paddle_build.sh` - `./paddle/phi/kernels/funcs/interpolate_function.h` - `./paddle/phi/kernels/funcs/strided_memcpy.h` - `./paddle/phi/core/utils/visit_place.h` - `./paddle/phi/core/flags.cc` - `./paddle/phi/backends/device_memory_aligment.h` - `./paddle/phi/backends/npu/npu_info.h` - `./paddle/phi/backends/context_pool.cc` - `./paddle/phi/backends/dynload/CMakeLists.txt` - `./paddle/phi/backends/dynload/dynamic_loader.cc`

8. 清理 `is_compiled_with_npu` 相关代码：

- `./tools/get_quick_disable_lt.py` - `./test/custom_kernel/custom_kernel_dot_c_setup.py` - `./test/custom_kernel/custom_kernel_dot_setup.py` - `./python/paddle/nn/functional/conv.py` - `./python/paddle/distributed/transpiler/collective.py` - `./python/paddle/distributed/collective.py` - `./python/paddle/distributed/launch/context/device.py` - `./python/paddle/distributed/ps/utils/collective_transpiler.py` - `./python/paddle/distributed/fleet/layers/mpu/mp_ops.py` - `./python/paddle/distributed/fleet/utils/hybrid_parallel_inference.py` - `./python/paddle/distributed/fleet/meta_parallel/sharding/group_sharded_utils.py` - `./python/paddle/distributed/fleet/launch.py` - `./python/paddle/distributed/fleet/meta_optimizers/sharding/offload_helper.py` - `./python/paddle/distributed/fleet/meta_optimizers/common.py` - `./python/paddle/distributed/fleet/meta_optimizers/sharding_optimizer.py` - `./python/paddle/distributed/fleet/launch_utils.py` - `./python/paddle/distributed/parallel.py` - `./python/paddle/hapi/model.py` - `./python/paddle/__init__.py` - `./python/paddle/utils/install_check.py` - `./python/paddle/fluid/dygraph/varbase_patch_methods.py` - `./python/paddle/fluid/tests/unittests/c_embedding_op_base.py` - `./python/paddle/fluid/tests/unittests/test_device.py` - `./python/paddle/fluid/tests/unittests/test_dist_base.py` - `./python/paddle/fluid/tests/unittests/eager_op_test.py` - `./python/paddle/fluid/tests/unittests/test_var_base.py` - `./python/paddle/fluid/framework.py` - `./python/paddle/fluid/__init__.py` - `./python/paddle/fluid/device_worker.py` - `./python/paddle/fluid/optimizer.py` - `./python/paddle/fluid/executor.py` - `./python/paddle/amp/grad_scaler.py` - `./python/paddle/static/nn/common.py` - `./python/paddle/static/amp/decorator.py` - `./python/paddle/static/amp/fp16_lists.py` - `./python/paddle/static/amp/amp_nn.py` - `./python/paddle/device/__init__.py` - `./paddle/fluid/pybind/pybind.cc`

9. 清理 `PADDLE_WITH_ASCEND` 相关代码，具体如下：

- `./paddle/fluid/operators/ascend_trigger_op.h` - `./paddle/fluid/pybind/tensor.cc` - `./paddle/fluid/pybind/ascend_wrapper_py.cc` - `./paddle/fluid/pybind/parallel_executor.cc` - `./paddle/fluid/pybind/pybind.cc` - `./paddle/fluid/pybind/place.cc`

10. 清理 `WITH_ASCEND` 相关代码，具体如下：

- `./CMakeLists.txt` - `./test/asp/CMakeLists.txt` - `./cmake/third_party.cmake` - `./cmake/configure.cmake` - `./cmake/external/warpctc.cmake` - `./cmake/external/protobuf.cmake` - `./cmake/external/threadpool.cmake` - `./cmake/external/gloo.cmake` - `./cmake/external/ascend.cmake` - `./python/paddle/fluid/tests/unittests/CMakeLists.txt` - `./python/paddle/fluid/tests/unittests/collective/CMakeLists.txt` - `./python/paddle/fluid/tests/unittests/collective/fleet/CMakeLists.txt` - `./python/paddle/fluid/tests/unittests/distributed_passes/CMakeLists.txt` - `./paddle/fluid/operators/ascend_trigger_op.h` - `./paddle/fluid/operators/CMakeLists.txt` - `./paddle/fluid/framework/fleet/CMakeLists.txt` - `./paddle/fluid/distributed/fleet_executor/test/CMakeLists.txt` - `./paddle/fluid/platform/device/CMakeLists.txt` - `./paddle/fluid/platform/device/npu/CMakeLists.txt` - `./paddle/fluid/inference/tests/api/CMakeLists.txt` - `./paddle/fluid/pybind/CMakeLists.txt`

11. 清理 paddle_build.sh 脚本中所有的 WITH_MLU WITH_ASCEND* 关键字

- `./paddle/scripts/paddle_build.sh`

KimBioInfoStudio commented 1 year ago

任务编号	任务内容	认领人	PR链接
4	清理 `is_compiled_with_mlu` 相关代码	@KimBioInfoStudio	#52378
8	清理 `is_compiled_with_npu` 相关代码	@KimBioInfoStudio	#52385
11	清理 `paddle_build.sh` 脚本中所有的 `WITH_MLU` `WITH_ASCEND*`关键字	@KimBioInfoStudio	#52373
12	清理 WITH_CNCL 相关代码	@KimBioInfoStudio	#52434
14	清理 dockerfile	@KimBioInfoStudio	#52474

遗留问题

代码冲突

[x] #52385 #52378 之间存在冲突, 任何一个先merge, 都需要大量工作 rebase 后一个
[x] #52385 / #52378 与其他子任务存在冲突
修改造成功能缺失
[ ] distibuted 相关修改 block PaddleCustomDevice distribued 功能，需要为 CustomDevice 重新设计 API https://github.com/PaddlePaddle/Paddle/issues/52319#issuecomment-1490259124
[x] ./python/paddle/distributed/fleet/layers/mpu/mp_ops.py 有残留
[x] ./python/paddle/distributed/fleet/meta_optimizers/common.py 有残留
[x] ./python/paddle/distributed/fleet/meta_optimizers/sharding_optimizer.py 有残留
[x] ./python/paddle/distributed/fleet/utils/hybrid_parallel_inference.py
[x] ./python/paddle/distributed/fleet/meta_parallel/sharding/group_sharded_utils.py
[x] ./python/paddle/distributed/fleet/meta_parallel/sharding/group_sharded_utils.py
[x] ./python/paddle/fluid/tests/unittests/c_embedding_op_base.py
[ ] ./paddle/fluid/pybind/pybind.cc 大概率失去 profiling 功能
暂时不能完全去掉 npu 相关代码
[x] ./python/paddle/nn/functional/conv.py
[x] ./python/paddle/fluid/dygraph/varbase_patch_methods.py
[x] ./python/paddle/fluid/framework.py
[x] ./python/paddle/fluid/__init__.py
[x] ./python/paddle/fluid/optimizer.py
[x] ./python/paddle/fluid/executor.py
[x] ./python/paddle/amp/grad_scaler.py
[x] ./python/paddle/static/nn/common.py
[x] ./python/paddle/static/amp/decorator.py
[x] ./python/paddle/static/amp/fp16_lists.py
[x] ./python/paddle/static/amp/amp_nn.py
[x] ./python/paddle/device/__init__.py

KimBioInfoStudio commented 1 year ago

@GreatV distributed 相关的部分代码不能直接删除，需要修改, 目前 PaddleCustomDevice 一些 distributed 相关功能并没有完全 decouple cc @luotao1 @qili93

KimBioInfoStudio commented 1 year ago

CC: @KevinYuK @yangulei

KimBioInfoStudio commented 1 year ago

seems cncl is mlu ccl, also need clean up @GreatV CC: @luotao1 @qili93

GreatV commented 1 year ago

@KimBioInfoStudio 确实缺少了WITH_CNCL的清理

mrcangye commented 1 year ago

任务编号 | 任务内容 | 认领人 | PR链接 -- | -- | -- | -- 1 | 删除 paddle/fluid/platform/device/mlu 目录|@mrcangye| #52382 9 | 清理 `PADDLE_WITH_ASCEND` 相关代码 | @mrcangye | #52403

Young-Flash commented 1 year ago

认领 2

KimBioInfoStudio commented 1 year ago

@KimBioInfoStudio 确实缺少了WITH_CNCL的清理

@GreatV 同理，还有 hccl

GreatV commented 1 year ago

@KimBioInfoStudio hccl这些应该会在清理 PADDLE_WITH_ASCEND_CL 被清理掉吧

mrcangye commented 1 year ago

paddle/fluid/platform/device/mlu/mlu_collective_helper.cc
paddle/fluid/platform/device/mlu/cncl_helper.h
paddle/fluid/platform/device/mlu/device_context_allocator.h
paddle/fluid/platform/device/mlu/device_context_test.cc

第一项任务中，这四个文件是不是也要删除

GreatV commented 1 year ago

paddle/fluid/platform/device/mlu/mlu_collective_helper.cc
paddle/fluid/platform/device/mlu/cncl_helper.h
paddle/fluid/platform/device/mlu/device_context_allocator.h
paddle/fluid/platform/device/mlu/mlu_collective_helper.cc

第一项任务中，这四个文件是不是也要删除

应该是的

jjyaoao commented 1 year ago

任务编号 | 任务内容 | 认领人 | PR链接 -- | -- | -- | -- 5 | 删除 `paddle/fluid/operators/*_npu.*`、清理 `WITH_ASCEND_INT64` 相关代码 | @jjyaoao | #52559 6 | 删除 `/paddle/fluid/platform/device/npu` 目录 | @jjyaoao | #52384

engineer1109 commented 1 year ago

任务编号 | 任务内容 | 认领人 | PR链接 -- | -- | -- | -- 7 | 清理 WITH_ASCEND_CL PADDLE_WITH_ASCEND_CL WITH_ASCEND_CXX11 相关代码 | @engineer| https://github.com/PaddlePaddle/Paddle/pull/52448 -- | -- | --

GreatV commented 1 year ago

任务编号	任务内容	认领人	PR链接
3	清理 `WITH_MLU` 相关代码	@GreatV	#52546

KimBioInfoStudio commented 1 year ago

paddle/fluid/platform/device/mlu/mlu_collective_helper.cc
paddle/fluid/platform/device/mlu/cncl_helper.h
paddle/fluid/platform/device/mlu/device_context_allocator.h
paddle/fluid/platform/device/mlu/device_context_test.cc

第一项任务中，这四个文件是不是也要删除

是的，现在 mlu 走的是 decvice / custom device, 所以 mlu 整个 dir 都可以删除

KimBioInfoStudio commented 1 year ago

@KimBioInfoStudio hccl这些应该会在清理 PADDLE_WITH_ASCEND_CL 被清理掉吧

应该不行，有些代码是在 Python 里面，WITH_ASCEND_CL 大部分都是在 cmake 或者 cc 里面

yangulei commented 1 year ago

should we also mv the tests at python/paddle/fluid/tests/unittests/mlu and python/paddle/fluid/tests/unittests/npu to the CustomDevice repo?

ronny1996 commented 1 year ago

paddle/fluid/platform/device/mlu/mlu_collective_helper.cc
paddle/fluid/platform/device/mlu/cncl_helper.h
paddle/fluid/platform/device/mlu/device_context_allocator.h
paddle/fluid/platform/device/mlu/device_context_test.cc

第一项任务中，这四个文件是不是也要删除

是的，paddle/fluid/platform/device/mlu 删除即可

ronny1996 commented 1 year ago

should we also mv the tests at python/paddle/fluid/tests/unittests/mlu and python/paddle/fluid/tests/unittests/npu to the CustomDevice repo?

just clean up

Liyulingyue commented 1 year ago

任务编号	任务内容	认领人	PR链接
13	清理 HCCL 相关代码	@Liyulingyue	#52535

KimBioInfoStudio commented 1 year ago

@luotao1 Pls add a subtask for docs change thx

KimBioInfoStudio commented 1 year ago

@luotao1 @qili93 @ronny1996 do we need rem graphcore ipu this time?

ronny1996 commented 1 year ago

@luotao1 @qili93 @ronny1996 do we need rem graphcore ipu this time?

ipu不需要删除哈

ronny1996 commented 1 year ago

@luotao1 Pls add a subtask for docs change thx

docs repo暂时不用删哈

KimBioInfoStudio commented 1 year ago

目前 develop branch 还能通过下面的方法找到部分 device 相关残留	CMD	`f1b6a76b56`	`e0c14fdf25`	`06ecc6d2d3`	`adc2b7459f`	`8f5eae47cd`
`grep -rni npu \\| grep -vi -P "input\\|onpush\\|onpull\\|unpub"`	744	744	677	649	568	255
`grep -rni mlu`	131	35	35	3	3	0
`grep -rni hccl`	8	8	0	0	0	0
`grep -rni ascend\\|grep -vi "ascending"`	107	107	36	23	23	23
`grep -rni cncl`	0	0	0	0	0	0

GreatV commented 1 year ago

任务编号	任务内容	认领人	PR链接
16	清理 ASCEND 相关代码	@GreatV	#53046

KimBioInfoStudio commented 1 year ago

@ronny1996 @qili93 seems we should remove cnpai for mlu profiler, add redesign a new api for all custom_device cc @luotao1

KimBioInfoStudio commented 1 year ago

@luotao1 @Liyulingyue in latest develop adc2b7459f there are still 3 mlu keyword in code

tools/infer_prune_patches/op_registry.h.patch:97:   REGISTER_OP_KERNEL(op_type, MLU, ::paddle::platform::MLUPlace, __VA_ARGS__)
tools/infer_prune_patches/op_registry.h.patch:131: #define REGISTER_OP_MLU_KERNEL_FUNCTOR(op_type, ...)                  \
tools/infer_prune_patches/op_registry.h.patch:133:       op_type, MLU, ::paddle::platform::MLUPlace, DEFAULT_TYPE,       \

luotao1 commented 1 year ago

in latest develop adc2b7459f there are still 3 mlu keyword in code

@KimBioInfoStudio @Liyulingyue 这个patch先不用改，是推理侧为了裁剪库体积写的脚本，不是简单删除即可。

KimBioInfoStudio commented 1 year ago

in latest develop adc2b74 there are still 3 mlu keyword in code

@KimBioInfoStudio @Liyulingyue 这个patch先不用改，是推理侧为了裁剪库体积写的脚本，不是简单删除即可。

Okay, 这个应该是需要重新取diff了，之前一致有个疑惑，删除custom device 代码会导致infer lib size 增加，现在大致猜测是因为 diff apply 不上去导致的

luotao1 commented 1 year ago

https://github.com/PaddlePaddle/Paddle/search?q=npu 还剩下最后一项【清理 NPU 相关代码】，还有不少npu代码。

luotao1 commented 1 year ago

@KimBioInfoStudio @Liyulingyue 整个 tools/infer_prune_patches 目录都可以删除，https://github.com/PaddlePaddle/Paddle/pull/53193#issuecomment-1534447963 已经使用另一种方式在裁剪预测库了。

KimBioInfoStudio commented 1 year ago

CMD	`b428e8f35c`
`grep -rni npu \\| grep -vi -P "input\\|onpush\\|onpull\\|unpub"`	255
`grep -rni mlu`	0
`grep -rni hccl`	0
`grep -rni ascend\\|grep -vi "ascending"`	23
`grep -rni cncl`	0

luotao1 commented 1 year ago

感谢 @KimBioInfoStudio 统计

grep -rni ascend|grep -vi "ascending"

ascend 关键字已检查完毕，剩余的都不需要删除

grep -rni npu | grep -vi -P "input|onpush|onpull|unpub"

53699 已合入，会再次减少
https://github.com/PaddlePaddle/Paddle/pull/53699#issuecomment-1549065303 还剩余tools路径和 cmake路径下确认下是不是 npu 的 dockerfile 以及 npu.cmake 等文件，这些都是可以删除的。 @Liyulingyue

luotao1 commented 1 year ago

昇腾和寒武纪相关代码退场已全部完成，感谢参与的小伙伴们！

感谢 @GreatV 拆分任务形成 Tracking issue； @KimBioInfoStudio 多次统计需要清理的内容； @Liyulingyue 完成最后的收尾工作，以及所有的小伙伴们！

按 merge 的时间顺序，排名不分先后： @KimBioInfoStudio (5) @mrcangye (2) @jjyaoao (3) @Young-Flash (1) @Liyulingyue (3) @engineer1109 (1) @GreatV (2)

欢迎继续参与快乐开源的其他任务！

PaddlePaddle / Paddle

昇腾和寒武纪相关代码退场 Tracking Issue #52319

遗留问题

代码冲突

修改造成功能缺失

暂时不能完全去掉 npu 相关代码

53699 已合入，会再次减少