[DRAFT] Remove duplicated lines

[alete89]

Naive approach that closes #6

• should add an arg to disable this behavior?
• is there any other consideration that prevent us to remove any duplicated line? (like, don't deduplicate if it's a comment)

[alete89]

:thinking:

[rehandalal]

i'm not sure this naive approach will work without mangling the file in more complex situations. i think the correct approach is to dedupe in the sort_requirements function before it reassembles the file. i think this problem is less trivial to solve than it seems on the surface. portions of the original script may need to change to accommodate this behaviour. it should also definitely be behind a flag perhaps something like --remove-duplicates.

[alete89]

I agree. I'll give it a new try later. Thanks.

[rehandalal]

thanks for attempting this! ill think about it some more and see if i can provide some better guidance on the original issue for an approach.