support dynamic re-initialization of PCI capabilities

[tmakatos]

Here's a list of questions we need to answer in order to properly implement capabilities:

• Can capabilities be added or removed dynamically? By the device or by the host?
• Can capability header be modified? How should a write by the host to the capability header be handled?
• Does the device need to handle access to the vendor-specific capability?

This might not be in the spec, QEMU might be a good place to start.

[jlevon]

Side note: PCI_CAP_ID_HT is another cap that can have multiple instances

[jlevon]

An example of the device needing a callback for a vendor cap:

 21 #define vsc_read(dev, offset, val) \                                             
 22         pci_read_config_dword((dev)->pdev, (dev)->vsc_addr + (offset), (val))    

 58         dev->vsc_addr = pci_find_capability(dev->pdev,                           
 59                                             PCI_CAP_ID_VNDR);                    

 64 int mlx5_vsc_gw_lock(struct mlx5_core_dev *dev)                                  
 65 {                                                                                
 66         u32 counter = 0;                                                         
 67         int retries = 0;                                                         
 68         u32 lock_val;                                                            
 69         int ret;                                                                 
 70                                                                                  
 71         pci_cfg_access_lock(dev->pdev);                                          
 72         do {                                                                     
 73                 if (retries > VSC_MAX_RETRIES) {                                 
 74                         ret = -EBUSY;                                            
 75                         goto pci_unlock;                                         
 76                 }                                                                
 77                                                                                  
 78                 /* Check if semaphore is already locked */                       
 79                 ret = vsc_read(dev, VSC_SEMAPHORE_OFFSET, &lock_val);            
 80                 if (ret)                                                         
 81                         goto pci_unlock;                                         
 82                                                                                  
 83                 if (lock_val) {                                                  
 84                         retries++;                                               
 85                         usleep_range(1000, 2000);                                
 86                         continue;                                                
 87                 }                                                                
 88                                                                                  
 89                 /* Read and write counter value, if written value is             
 90                  * the same, semaphore was acquired successfully.                
 91                  */                                                              
 92                 ret = vsc_read(dev, VSC_COUNTER_OFFSET, &counter);               
 93                 if (ret)                                                         
 94                         goto pci_unlock;                                         
 95                                                                                  
 96                 ret = vsc_write(dev, VSC_SEMAPHORE_OFFSET, counter);             
 97                 if (ret)                                                         
 98                         goto pci_unlock;                                         
 99                                                                                  
100                 ret = vsc_read(dev, VSC_SEMAPHORE_OFFSET, &lock_val);            
101                 if (ret)                                                         
102                         goto pci_unlock;                                         
103                                                                                  
104                 retries++;                                                       
105         } while (counter != lock_val);                                           
106                                                                                  
107         return 0;                                                                
108                                                                                  
109 pci_unlock:                                                                      
110         pci_cfg_access_unlock(dev->pdev);                                        
111         return ret;                                                              
112 }                                                                                

[jlevon]

Progress on this based on our current understanding:

• we completely disallow any host (client) writes to cap headers
• API users definitely need callback facility for vendor caps, now supported
• caps cannot be added/removed by the host (client)
• caps are designed as initial config mechanisms, therefore we could reasonably expect them to not need to by dynamic. The open question here is if there are devices we care about that re-initialize themselves to the point they'd need to reset the cap list, and if so how we could support it.