search

devopshq / tfs

Microsoft TFS API Python client
https://devopshq.github.io/tfs/
MIT License
110 stars 28 forks source link

Library types are too heavy to work with #71

Closed orihomie closed 4 years ago

orihomie commented 5 years ago

I was able to do some kind of task to select about 2.5 thousands workitems (by work item query), and there was one big problem I was tryin to solve - to parse them ASAP.

And that was very ridicilous experience:

1) my idea was to parse them asynchronously and that taken about 2-3 hours for me to get the problem: heavy workitems are hardly to parse asynchronously on multiprocessing pool.

2) And the second idea was to take only fields I've needed then parse them on multiproc pool and that worked perfectly for about 20 seconds on the main processor without any pools.

This is where I've got an idea - to make workitems more 'lighter'. For now I dont know how to make it real, only thought to make it some kind of Lazy from C#, but could anyone give a suggestion if its real or not and if it's ok to deal with such tasks as I did?

Boltyk commented 5 years ago

@orihomie C#?

What are you doing with all these work items? What method are you using to get these 2.5 grans of work items? run_wiql?

orihomie commented 5 years ago

@Boltyk yes, run_wiql, that task was sort of statistics.

I've said kind of C# Lazy type where you can get parts of object not instantly on initialization of object, but on demand of each part you want to get. And this is only idea, I dont know for now if it's good or bad.

Boltyk commented 5 years ago

Not sure I've got the problem.

So you run_wiql and it gives you list of work items. This list is just a list of work item ids, as I understand. https://docs.microsoft.com/en-us/azure/devops/integrate/previous-apis/wit/wiql?view=tfs-2015#get-work-items Supposed to be pretty lightweight

Then you try to get them one by one for some processing. And there you hit the problem, right?

Was interested by myself about laziness in Python recently. Found this article http://derrickgilland.com/posts/late-value-passing-for-lazy-method-chaining-in-python/

orihomie commented 5 years ago

No, it's not just ids, after wiql runs every id turns into WorkItem-type instance.

Boltyk commented 5 years ago

Ough, just having list of these work items as a python workitem objects in a one giant list is a problem itself... Interesting

orihomie commented 5 years ago

It is possible to run that id-workitem casting asynchronously, that would be a bit speed up, but that's not for this issue.

orihomie commented 5 years ago

I was talking about this kind of laziness: https://pypi.org/project/lazy/

allburov commented 5 years ago
  1. And the second idea was to take only fields I've needed then parse them on multiproc pool and that worked perfectly for about 20 seconds on the main processor without any pools.

The issue https://github.com/devopshq/tfs/issues/14 can help you get a "light" WI.

Boltyk commented 4 years ago

@orihomie did you get a workaround?