search

microsoft / ebpf-for-windows

eBPF implementation that runs on top of Windows
MIT License
2.94k stars 238 forks source link

Native image backwards compatibility #3863

Closed shankarseal closed 1 month ago

shankarseal commented 1 month ago

Discussed in https://github.com/microsoft/ebpf-for-windows/discussions/3839

Originally posted by **lmb** September 17, 2024 I recently tried to load a .sys file compiled against roughly v0.18 against current main. This failed due to a mismatch of the hash of the program info. It seems like upgrading the runtime broke the old native image. Was this breakage intentional? What is the plan for backwards compatibility in the future? Having to always load the .sys against the correct version of the runtime would be very limiting.
shpalani commented 1 month ago

Was this breakage intentional? What is the plan for backward compatibility in the future? Having to always load the .sys against the correct version of the runtime would be very limiting.

We aren't aware of any backward compatibility breakage. Can you please share your native program file to help repro from our side or provide the logs with errors observed with logman traces? What can of extension is being used in your program?

Also, did you try, stop and start the ebpf core binaries? sc.exe stop ebpfcore sc.exe start ebpfcore sc.exe stop netebpfext sc.exe start netebpfext [and ebpfsvc if non-jit image]

shpalani commented 1 month ago

@lmb Any update?

lmb commented 1 month ago

Something very weird is going on which I can't explain. Here is a sanitized trace event log output of a failing load:

EbpfSuccess              ebpf_api_initiate returned success                                               

EbpfGenericMessage       ebpf_object_load_native_fds                                                      

EbpfGenericMessage       _ebpf_object_load_native                                                         

EbpfGenericMessage       _ebpf_object_load_native                                                         
        *guid1={d65f1d4b-9d0d-43ff-b3b2-18ab5688a7f0} *guid2={61513726-f4b3-4746-9b80-0a684dabd63f} string=testdata/printk.sys
EbpfGenericMessage       _load_native_module                                                              

EbpfGenericMessage       _ebpf_epoch_activate_cpu                                                         

EbpfGenericMessage       Activating CPU                                                                   
        value=2
EbpfGenericMessage                                                                                        
        Exit=_ebpf_epoch_activate_cpu
EbpfGenericMessage       _ebpf_core_protocol_load_native_module                                           

EbpfGenericMessage       ebpf_native_load                                                                 

EbpfGenericMessage       _ebpf_native_client_attach_callback: Called for                                  
        *guid={61513726-f4b3-4746-9b80-0a684dabd63f}
EbpfSuccess              _ebpf_native_provider_attach_client_callback returned success                    

EbpfApiError             \Registry\Machine\System\CurrentControlSet\Services\d65f1d4b-9d0d-43ff-b3b2-18ab5688a7f0 Api=ZwLoadDriver status=0x0
EbpfGenericMessage       ebpf_handle_create                                                               

EbpfGenericMessage       ebpf_handle_create: returning handle                                             
        value=508
EbpfSuccess              ebpf_handle_create returned success                                              

EbpfSuccess              ebpf_native_load returned success                                                

EbpfSuccess              _ebpf_core_protocol_load_native_module returned success                          

EbpfSuccess              invoke_ioctl returned success                                                    

EbpfSuccess              _load_native_module returned success                                             

EbpfGenericMessage       _load_native_programs                                                            

EbpfGenericMessage       _ebpf_core_protocol_load_native_programs                                         

EbpfGenericMessage       ebpf_native_get_count_of_maps                                                    

EbpfSuccess              ebpf_native_get_count_of_maps returned success                                   

EbpfGenericMessage       ebpf_native_get_count_of_programs                                                

EbpfSuccess              ebpf_native_get_count_of_programs returned success                               

EbpfGenericMessage       ebpf_native_load_programs                                                        

EbpfGenericMessage       _ebpf_native_create_maps                                                         

EbpfSuccess              _ebpf_native_create_maps returned success                                        

EbpfGenericMessage       ebpf_program_create_and_initialize                                               

EbpfGenericMessage       ebpf_program_create                                                              

EbpfGenericMessage       Program information provider module ID mismatch.                                 

EbpfGenericMessage       Program information provider module ID mismatch.                                 

EbpfGenericMessage       Program information provider module ID mismatch.                                 

EbpfGenericMessage       Program information provider module ID mismatch.                                 

EbpfGenericMessage       Program information provider module ID mismatch.                                 
        *guid1={608c517c-6c52-4a26-b677-bb1c34425adf} *guid2={8d2a1d3f-9ce6-473d-b48e-17aa5c5581fe}       
EbpfGenericMessage       Program information provider module ID mismatch.                                 
        *guid1={608c517c-6c52-4a26-b677-bb1c34425adf} *guid2={ce8ccef8-4241-4975-984d-bb3921dfa73c}       
EbpfGenericMessage       ebpf_duplicate_program_data                                                      

EbpfGenericMessage       ebpf_duplicate_program_info                                                      

EbpfSuccess              ebpf_duplicate_program_info returned success                                     

EbpfSuccess              ebpf_duplicate_program_data returned success                                     

NetEbpfExtGenericMessage                                                                                  
        Enter=_net_ebpf_extension_program_info_provider_attach_client
NetEbpfExtGenericMessage _net_ebpf_extension_program_info_provider_attach_client returned success         

EbpfGenericMessage       Program information provider module ID mismatch.                                 
        *guid1={608c517c-6c52-4a26-b677-bb1c34425adf} *guid2={92ec8e39-aeec-11ec-9a30-18602489beee}       
EbpfGenericMessage       Program information provider module ID mismatch.                                 
        *guid1={608c517c-6c52-4a26-b677-bb1c34425adf} *guid2={43fb224d-68f8-46d6-aa3f-c856518cbb32}       
EbpfGenericMessage       eBPF object initialized                                                          
        object=0xFFFF800ECD87B7B0                     object_type=       3
EbpfSuccess              ebpf_program_create returned success                                             

EbpfGenericMessage       ebpf_handle_create                                                               

EbpfGenericMessage       ebpf_handle_create: returning handle                                             
        value=512
EbpfSuccess              ebpf_handle_create returned success                                              

EbpfSuccess              ebpf_program_create_and_initialize returned success                              

EbpfGenericMessage       ebpf_core_load_code                                                              

EbpfGenericMessage       ebpf_program_load_code                                                           

EbpfGenericMessage       _ebpf_program_load_machine_code                                                  

EbpfGenericMessage       ebpf_unmap_memory                                                                

EbpfGenericMessage                                                                                        
        Exit=ebpf_unmap_memory
EbpfSuccess              _ebpf_program_load_machine_code returned success                                 

EbpfSuccess              ebpf_program_load_code returned success                                          

EbpfSuccess              ebpf_core_load_code returned success                                             

EbpfGenericMessage       _ebpf_native_resolve_maps_for_program                                            

EbpfSuccess              _ebpf_native_resolve_maps_for_program returned success                           

EbpfGenericMessage       _ebpf_native_resolve_helpers_for_program                                         

EbpfGenericMessage       ebpf_core_resolve_helper                                                         

EbpfGenericMessage       ebpf_program_set_helper_function_ids                                             

EbpfSuccess              ebpf_program_set_helper_function_ids returned success                            

EbpfGenericMessage       ebpf_program_get_helper_function_addresses                                       

EbpfGenericMessage       _ebpf_program_get_helper_function_address                                        

EbpfSuccess              _ebpf_program_get_helper_function_address returned success                       

EbpfGenericMessage       _ebpf_program_get_helper_function_address                                        

EbpfSuccess              _ebpf_program_get_helper_function_address returned success                       

EbpfGenericMessage       _ebpf_program_get_helper_function_address                                        

EbpfSuccess              _ebpf_program_get_helper_function_address returned success                       

EbpfGenericMessage       _ebpf_program_get_helper_function_address                                        

EbpfSuccess              _ebpf_program_get_helper_function_address returned success                       

EbpfGenericMessage       _ebpf_program_get_helper_function_address                                        

EbpfSuccess              _ebpf_program_get_helper_function_address returned success                       

EbpfSuccess              ebpf_program_get_helper_function_addresses returned success                      

EbpfGenericMessage       ebpf_program_set_program_info_hash                                               

EbpfGenericMessage       _ebpf_program_compute_program_information_hash                                   

EbpfSuccess              _ebpf_program_compute_program_information_hash returned success                  

EbpfGenericError         ebpf_program_set_program_info_hash returned error                                
        Error=6
EbpfGenericMessage       ebpf_program_clear_helper_function_ids                                           

EbpfGenericError         ebpf_core_resolve_helper returned error                                          
        Error=6
EbpfGenericMessage       ebpf_core_resolve_helper failed                                                  
        *guid={61513726-f4b3-4746-9b80-0a684dabd63f}
EbpfGenericError         _ebpf_native_resolve_helpers_for_program returned error                          
        Error=6
EbpfGenericMessage       ebpf_handle_close                                                                

EbpfGenericMessage       ebpf_handle_close: closing handle                                                
        value=512
EbpfGenericMessage       _ebpf_program_zero_ref_count                                                     

EbpfGenericMessage       _ebpf_program_detach_links                                                       

EbpfGenericMessage                                                                                        
        Exit=_ebpf_program_detach_links
EbpfGenericMessage                                                                                        
        Exit=_ebpf_program_zero_ref_count
EbpfSuccess              ebpf_handle_close returned success                                               

EbpfGenericMessage       ebpf_native_load_programs: program load failed                                   
        *guid={61513726-f4b3-4746-9b80-0a684dabd63f}
EbpfGenericMessage                                                                                        
        Exit=ebpf_native_release_reference
EbpfGenericError         ebpf_native_load_programs returned error                                         
        Error=6
EbpfGenericError         _ebpf_core_protocol_load_native_programs returned error                          
        Error=6
EbpfApiError                                                                                              
        Api="ebpf_core_invoke_protocol_handler"      status=0xC000000D
EbpfApiError                                                                                              
        Api=DeviceIoControl                          last_error=0x57
EbpfGenericError         invoke_ioctl returned error                                                      
        Error=87
EbpfApiError                                                                                              
        Api=invoke_ioctl                             last_error=0x57 module_id={61513726-f4b3-4746-9b80-0a684dabd63f}
EbpfGenericError         _load_native_programs returned error                                             
        Error=6
EbpfGenericMessage       _ebpf_object_load_native: load native programs failed                            
        string_value=testdata/printk.sys
EbpfGenericMessage                                                                                                Exit=ebpf_native_release_reference
EbpfGenericMessage       _ebpf_native_client_detach_callback: Called for                                          *guid={61513726-f4b3-4746-9b80-0a684dabd63f}
EbpfGenericMessage                                                                                                Exit=ebpf_native_release_reference
EbpfSuccess              _ebpf_native_provider_detach_client_callback returned success                    

EbpfGenericMessage       eBPF object terminated                                                                   object=0xFFFF800ECD87B7B0 object->type=       3
EbpfGenericMessage       _ebpf_program_free                                                               

EbpfGenericMessage       _ebpf_epoch_activate_cpu                                                         

EbpfGenericMessage       Activating CPU                                                                           value=1
EbpfGenericMessage                                                                                                Exit=_ebpf_epoch_activate_cpu
EbpfGenericMessage       _ebpf_epoch_deactivate_cpu                                                       

EbpfGenericMessage       Deactivating CPU                                                                         value=2
EbpfGenericMessage                                                                                                Exit=_ebpf_epoch_deactivate_cpu
EbpfGenericMessage       _ebpf_epoch_deactivate_cpu                                                       

EbpfGenericMessage       Deactivating CPU                                                                         value=1
EbpfGenericMessage                                                                                                Exit=_ebpf_epoch_deactivate_cpu
EbpfGenericMessage       ebpf_native_release_reference: ref is 0, complete detach callback                        *guid={61513726-f4b3-4746-9b80-0a684dabd63f}
EbpfGenericMessage                                                                                                Exit=ebpf_native_release_reference
EbpfGenericMessage       ebpf_free_trampoline_table                                                       

EbpfGenericMessage                                                                                                Exit=ebpf_free_trampoline_table
EbpfGenericMessage                                                                                                Exit=_ebpf_program_free
EbpfGenericMessage       _ebpf_object_load_native: Deleting service                                               wstring=d65f1d4b-9d0d-43ff-b3b2-18ab5688a7f0
EbpfGenericError         _ebpf_object_load_native returned error                                                  Error=6
EbpfGenericError         ebpf_object_load_native_fds returned error                                               Error=6
exit status 1

ebpf_object_load_native_fds is new code available in https://github.com/lmb/ebpf-for-windows/commit/af1697441e071b421140cc45ed20f7f41d08fddd (there are more changes in that branch).

I believe the error is in EbpfGenericError ebpf_program_set_program_info_hash returned error Error=6 generated here: https://github.com/microsoft/ebpf-for-windows/blob/d712e97fd7baf62b13b6270fbdf3a78c69038d93/libs/execution_context/ebpf_program.c#L1881-L1888

The problem goes away after:

  1. Uninstall runtime
  2. Blow away ./x64/
  3. Rebuild on command line (debug build)
  4. Install runtime
  5. Use printk.sys from new build

Here is a zip containing the two modules: printk.zip

shpalani commented 1 month ago

The problem is not reproducible locally. Works as expected. Here are the steps performed.

  1. Installed ebpf-for-windows msi from NativeOnlyRelease image built locally in the latest 'main' repro, in a vm.
  2. v0.18.0: Got the Build-x64-native-only-NativeOnlyRelease's printk.sys. driver.
  3. Loaded the v0.18.0's printk.sys program with services running in step1: 
    
C:\Users\TDPUser\Downloads>netsh ebpf add program printk.sys
Loaded with ID 4

C:\Users\TDPUser\Downloads>

C:\Users\TDPUser\Downloads>netsh ebpf show program

ID  Pins  Links  Mode       Type           Name

====== ==== ===== ========= ============= ==================== 4 1 1 NATIVE bind func

C:\Users\TDPUser\Downloads>


4. The printk.sys driver from v0.18.0 was loaded successfully with the main's ebpf engine, hence it is backward compatable.
shpalani commented 1 month ago

@lmb : Please let us know if we can close this issue. There is no backward compatibility issue with printk.sys. Pasted the logs above.