search

humanmade / hm-content-import

Migration framework for WordPress, attempts to reduce overhead in migrating content from differing data sources
GNU General Public License v3.0
27 stars 7 forks source link

Iterator that supports multiple items within files #56

Open johnbillion opened 5 years ago

johnbillion commented 5 years ago

Using a separate file for each individual exported item is fine until you get to really large numbers and want to batch multiple items into a file in order to reduce the overall number of files (for I/O performance purposes, for example).

I'm working on writing an iterator that supports a JSON encoded array within each file that gets iterated over by the Files iterator, including support for updating the progress bar correctly, and only passing one item to process_item() as expected.

Combined with #55 this allows for much more efficient use of files on the filesystem.

tcrsavage commented 5 years ago

@johnbillion I want to pull out the progress indicator moving forward. It's not particularly useful for a developer to see a progress bar when that information can be contained within a line by line debug e.g:

1/1000 Processing item source ID: 123
1/1000 Inserted item ID: 456
2/1000 Processing item source ID: 789
2/1000 Warning: Unable to insert item 789