Closed yihe9063 closed 4 years ago
Seems you are right - and we may need to add ...
to generic create_vocabulary
. PR is welcome.
Just started my vacation, can take a look when I got back.
@yihe9063 any chance for you to send PR to fix this?
Hi, (This will be my main account.)
Sry for a very late reply, I got caught in two separate vacation trips, and the new virus situation.
I can try to fix it in the upcoming weekends, since I have never collab on a open source project before, it may take some time to get into it.
By the mean time I discovered that itoken_parallel was disabled for Windows in https://github.com/dselivanov/text2vec/issues/293 Any reason why "itoken_par" (not "ifiles_par") would not work in Windows?
itoken_parallel = function(iterable, ...) {
if(.Platform$OS.type != "unix") {
warning("`ifiles_parallel` is not supported on windows - falling back to `itoken`")
UseMethod("itoken")
} else {
UseMethod("itoken_parallel")
}
}
Also the warning msg was wrong for the itoken_parallel, still using the same msg for "ifiles_par" I could try to fix the warning msg at the same time.
But since direct parallel processing was disabled on Windows, is the best practice now to use some future_map alike functionality to do distribute the load on Windows?
Update: I just read the docs for create_dtm, looks like if we split the load to a few iterators, the function should still compute the DTM in parallel, will test whether this is the case.
Update2: got an error when tried that way:
Error in UseMethod("create_dtm") :
no applicable method for 'create_dtm' applied to an object of class "list"
Maybe I should open another issue?
Best regards, Ian (Yiyang)
But since direct parallel processing was disabled on Windows, is the best practice now to use some future_map alike functionality to do distribute the load on Windows?
Update: I just read the docs for create_dtm, looks like if we split the load to a few iterators, the function should still compute the DTM in parallel, will test whether this is the case.
I theory yes, but I don't recommend to do that.
I'm closing as the issue is not relevant anymore.
Hi,
Version: 0.5.1, OS: Windows
Is it possible that the extra arguments "..." for create_vocabulary are not catched?
As I could see in the source code for 0.6:
Maybe it should automatically use the class method create_vocabulary.itoken_parallel which has the "...". However I couldnt pass the ".export" in any way to create_vocabulary with an initialized itoken_parallel, and got the "unused argument..." error.
Also for create_dtm I could pass .export to it without any problem.
Just saw the update notes for 0.6 could it be related?
Best regards, Yiyang (y-he2)