ericvaandering
commented
10 years ago Author: giffels Currently I have implemented input validation for list/insertDataTiers and insertBlocks. Still missing are regular expressions to validate the input strings. This can be easily added using the decorator for list APIs and the validateStringInput function for insert APIs. For the future it might be reasonable to merge them into one function. I have attached a patch to this ticket. Yuyi could you please review and check that insertBlock and insertDataTier is working properly? I have currently some problems with the unittest environment for the server.
I have also attached the code of DBSValidation.py inline, since performance is really important here. I currently fear that I did not found the optimal solution for validating the json input. Currently I am using recursion with a maximal depth of 6, which avoids code duplication and results in short code (validateJSONInput) to validate the input keys. I would appreciate any comments that makes validation faster.
Concerning performance. I got a worst case scenario for insertBlocks from Yuyi. The file (/afs/cern.ch/user/y/yuyi/public/block-data-sample/block.json) is about 46M. Reading and decoding the file using cjson takes approximately 4 seconds. The current validation takes approximately 12 seconds. The block contains 1000 files with 1000 lumi sections each.
from dbs.utils.dbsExceptionHandler import dbsExceptionHandler
from dbs.utils.dbsException import dbsException,dbsExceptionCode
def SQLInputValidation(input_data):
pass
def doSQLInputValidation(func):
def new_func(input_data):
SQLInputValidation(input_data)
return func(input_data)
return new_func
validArgs = {
################
'listDataTiers':['data_tier_name']
################
}
requiredInputKeys = {
################
'insertDataTier':['data_tier_name'],
################
'insertBlock':['file_conf_list', 'dataset_conf_list', 'block_parent_list', 'physics_group_name', 'processing_era', 'dataset', 'block', 'acquisition_era', 'primds', 'ds_parent_list', 'files', 'file_parent_list'],
################
'file_conf_list':['release_version', 'pset_hash', 'lfn', 'app_name', 'output_module_label'],
################
'dataset_conf_list':['release_version', 'pset_hash', 'app_name', 'output_module_label'],
################
'physics_group_name':[],
################
'processing_era':['processing_version'],
################
'dataset':['dataset', 'is_dataset_valid', 'physics_group_name', 'processed_ds_name', 'dataset_access_type', 'data_tier_name'],
################
'block': ['block_name', 'open_for_writing', 'origin_site_name'],
################
'acquisition_era':['acquisition_era_name'],
################
'primds':['primary_ds_type', 'primary_ds_name'],
################
'files':['check_sum', 'file_lumi_list', 'event_count', 'file_type', 'logical_file_name', 'file_size'],
################
'file_lumi_list':['lumi_section_num', 'run_num']
################
}
@doSQLInputValidation
def validateDataTiers(input_data):
if input_data.has_key('data_tier_name') and type(input_data['data_tier_name']) is not str:
dbsExceptionHandler('dbsException-invalid-input', 'data_tier_name given is not valid')
return input_data
def validateJSONInput(input_key,input_data):
if isinstance(input_data,dict):
return_data = {}
for key in requiredInputKeys[input_key]:
try:
return_data[key] = validateJSONInput(key,input_data[key])
except KeyError as ke:
dbsExceptionHandler('dbsException-invalid-input', "Invalid input")
elif isinstance(input_data,list):
return_data = [validateJSONInput(input_key,x) for x in input_data]
elif isinstance(input_data,str):
return_data = validateStringInput(input_data)
elif isinstance(input_data,int):
return input_data
else:
dbsExceptionHandler('dbsException-invalid-input', "Invalid input")
return return_data
def validateStringInput(input_key,input_data):
#Needs some checks for strings
return input_data
Original TRAC ticket 1949 reported by giffels Implementation of input validation for DBS3 as proposed in #499