vehicle_purch_no_cost_flag: limit ownership

pro100olga commented 4 years ago

For vehicle_purch_no_cost_flag consider only vehicles, where rights section has key of declarant ('1') or family (['step_2'].keys()) and at the same time ownershipType is one of the following: 'Власність', 'Спільна власність'

Smth like this:

for k in decl['declaration']['unified_source']['step_6'].keys():

    for right in decl['declaration']['unified_source']['step_6'][k]['rights'].keys():

        if right in list(decl['declaration']['unified_source']['step_2'].keys()) + ['1']:

            if decl['declaration']['unified_source']['step_6'][k]['rights'][right]['ownershipType'] in ['Власність', 'Спільна власність']:

                # this object should be considered, including also requirement for owning date andcost missing

Final description of the flag (written before closing issue): Check all transport objects. Raise the flag if there is an object for which:

date of owning is during the reported year AND
declarant or family member has right (Власність \ Спільна власність) for this object AND
cost of the object is not provided or zero

dchaplinsky commented 4 years ago

Should be fixed now.

pro100olga commented 4 years ago

Checked on 500K documents from annual 2017 decl. ~250 mismatches

False, but should be True: nacp_e6ef879e-d0b4-4271-b4a4-0fa4d9ee6dde nacp_e2271a8b-aa67-43d7-8160-eb883657fc34 nacp_f0ced39f-0d3e-4d83-9083-5b28f93c5798

True, but should be False: nacp_48926224-85bf-4e5a-bfde-697f69f91a86 nacp_bf4dadb2-a192-44a6-baba-4284f6bc001b nacp_ce1c27b1-61f2-4181-9dda-995fbec47e64

Do you check ownership only for declarant + family? Cause in these 3 last cases owner is a third party (type - Власність, but owner is not declarant or family member)

dchaplinsky commented 4 years ago

Hmm, again it's correct on my side. Will regenerate file and update you shortly.

On Wed, Apr 1, 2020 at 9:17 PM Olga Makarova notifications@github.com wrote:

Checked on 500K documents from annual 2017 decl. ~250 mismatches

False, but should be True: nacp_e6ef879e-d0b4-4271-b4a4-0fa4d9ee6dde nacp_e2271a8b-aa67-43d7-8160-eb883657fc34 nacp_f0ced39f-0d3e-4d83-9083-5b28f93c5798

True, but should be False: nacp_48926224-85bf-4e5a-bfde-697f69f91a86 nacp_bf4dadb2-a192-44a6-baba-4284f6bc001b nacp_ce1c27b1-61f2-4181-9dda-995fbec47e64

Do you check ownership only for declarant + family? Cause in these 3 last cases owner is a third party (type - Власність, but owner is not declarant or family member)

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/excieve/dragnet/issues/46#issuecomment-607412032, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABAA4R7ESKDWANKGO357ZDRKOAK3ANCNFSM4LTW7B3Q .

pro100olga commented 4 years ago

Checked mismatches mentioned above - all ok Will check on full dataset later

pro100olga commented 4 years ago

Checked on ~700K of annual declarations ~170 mismatches, all True, but should be False:

nacp_4b736266-e107-44fc-80c6-bd95a883f178 nacp_00c2c562-7c32-411d-a4f2-f5a8551679e7 nacp_07c0c9da-7462-4c67-bbd1-a4d6ee1a94b3

Pls check if you consider ownership only for declarant and family Cause in these cases owner is a third party (type - Власність, but owner is not declarant or family member)

dchaplinsky commented 4 years ago

Okay, I've fixed a function that was used to validate ownership. Previously it checked all the rights on the vehicle/real estate, and if there was at least one right of ownership/coownership it said it's okay. But in those cases above there are ownership/coownership rights for third persons (while they was declared by declarant or his/her family). I've made amendments to that function, it might affect following flags: has_foreign_real_estate estate_purch_no_cost estate_has_hidden_cost has_vehicle vehicle_purch_no_cost has_luxury_cars_v2

On Tue, Apr 14, 2020 at 11:13 PM Olga Makarova notifications@github.com wrote:

Do you mean, that both should be hidden (AND clause, not OR)?

Yes, please

Checked mismatches from 1st set - all ok.

Second set. nacp_4b389515-04bc-4271-8660-af03cf464b4d - this one should be resolved by adding cost assessment check (AND) nacp_ae44918a-367c-40e4-8af4-48dba87544e6 - why do you think it should be True? There is no object acquired in 2017 (but there is one where date is hidden - however I do not think we should count it)

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/excieve/dragnet/issues/45#issuecomment-613657565, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABAA4U3T4MO6UIIE7ZJRKLRMS7XHANCNFSM4LTWZVZQ .

On Wed, Apr 15, 2020 at 10:25 PM Olga Makarova notifications@github.com wrote:

Checked on ~700K of annual declarations ~170 mismatches, all True, but should be False:

nacp_4b736266-e107-44fc-80c6-bd95a883f178 nacp_00c2c562-7c32-411d-a4f2-f5a8551679e7 nacp_07c0c9da-7462-4c67-bbd1-a4d6ee1a94b3

Pls check if you consider ownership only for declarant and family Cause in these cases owner is a third party (type - Власність, but owner is not declarant or family member)

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/excieve/dragnet/issues/46#issuecomment-614234184, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABAA4VF3YUJSFC6CXMHKMLRMYC3FANCNFSM4LTW7B3Q .

pro100olga commented 4 years ago

Checked on 500K documents (annual declarations from 2017) using this code

All ok. Closed.

Final description of the flag: Check all transport objects. Raise the flag if there is an object for which:

date of owning is during the reported year AND
declarant or family member has right (Власність \ Спільна власність) for this object AND
cost of the object is not provided or zero

excieve / dragnet

vehicle_purch_no_cost_flag: limit ownership #46