Two other questions in addition to this...
Why does it download the nltk_data punkt tokenizers every time
AND
Can this be made to write a new sentence without being given the dataset each time? In other words, can I have it read HarryPotter(large).txt just once keep feeding it different fragments to make sentences out without having to fit it with the same data over and over?
(venv) ANTHONYs-iMac:t2 anthonycelio$ python3 main.py -ts "today i will" -tsc 10 -td datasets/harrypotterxxl.txt
[nltk_data] Downloading package punkt to
[nltk_data] /Users/anthonycelio/nltk_data...
[nltk_data] Unzipping tokenizers/punkt.zip.
Test sequence generation size changed to: 10
Training data load locaiton changed to: "datasets/harrypotterxxl.txt"
Loading text data from: (datasets/harrypotterxxl.txt)
Data load successful. WordCount: 15829
Beginning sentence structure parse...
Data normalised successful...
Beginning sentence vocabulary parse...
Data successfully fitted to the sentence structure network.
Vectors: 15826
No training data for vocab identifier: SYM
No training data for vocab identifier: $
No training data for vocab identifier: POS
No training data for vocab identifier: ``
No training data for vocab identifier: LS
Data successfully fitted to the vocabulary network.
Vectors: 15881
/Users/anthonycelio/t2/venv/lib/python3.5/site-packages/sklearn/utils/validation.py:395: DeprecationWarning: Passing 1d arrays as data is deprecated in 0.17 and will raise ValueError in 0.19. Reshape your data either using X.reshape(-1, 1) if your data has a single feature or X.reshape(1, -1) if it contains a single sample.
DeprecationWarning)
/Users/anthonycelio/t2/venv/lib/python3.5/site-packages/sklearn/utils/validation.py:395: DeprecationWarning: Passing 1d arrays as data is deprecated in 0.17 and will raise ValueError in 0.19. Reshape your data either using X.reshape(-1, 1) if your data has a single feature or X.reshape(1, -1) if it contains a single sample.
DeprecationWarning)
/Users/anthonycelio/t2/venv/lib/python3.5/site-packages/sklearn/utils/validation.py:395: DeprecationWarning: Passing 1d arrays as data is deprecated in 0.17 and will raise ValueError in 0.19. Reshape your data either using X.reshape(-1, 1) if your data has a single feature or X.reshape(1, -1) if it contains a single sample.
DeprecationWarning)
/Users/anthonycelio/t2/venv/lib/python3.5/site-packages/sklearn/utils/validation.py:395: DeprecationWarning: Passing 1d arrays as data is deprecated in 0.17 and will raise ValueError in 0.19. Reshape your data either using X.reshape(-1, 1) if your data has a single feature or X.reshape(1, -1) if it contains a single sample.
DeprecationWarning)
/Users/anthonycelio/t2/venv/lib/python3.5/site-packages/sklearn/utils/validation.py:395: DeprecationWarning: Passing 1d arrays as data is deprecated in 0.17 and will raise ValueError in 0.19. Reshape your data either using X.reshape(-1, 1) if your data has a single feature or X.reshape(1, -1) if it contains a single sample.
DeprecationWarning)
/Users/anthonycelio/t2/venv/lib/python3.5/site-packages/sklearn/utils/validation.py:395: DeprecationWarning: Passing 1d arrays as data is deprecated in 0.17 and will raise ValueError in 0.19. Reshape your data either using X.reshape(-1, 1) if your data has a single feature or X.reshape(1, -1) if it contains a single sample.
DeprecationWarning)
/Users/anthonycelio/t2/venv/lib/python3.5/site-packages/sklearn/utils/validation.py:395: DeprecationWarning: Passing 1d arrays as data is deprecated in 0.17 and will raise ValueError in 0.19. Reshape your data either using X.reshape(-1, 1) if your data has a single feature or X.reshape(1, -1) if it contains a single sample.
DeprecationWarning)
/Users/anthonycelio/t2/venv/lib/python3.5/site-packages/sklearn/utils/validation.py:395: DeprecationWarning: Passing 1d arrays as data is deprecated in 0.17 and will raise ValueError in 0.19. Reshape your data either using X.reshape(-1, 1) if your data has a single feature or X.reshape(1, -1) if it contains a single sample.
DeprecationWarning)
/Users/anthonycelio/t2/venv/lib/python3.5/site-packages/sklearn/utils/validation.py:395: DeprecationWarning: Passing 1d arrays as data is deprecated in 0.17 and will raise ValueError in 0.19. Reshape your data either using X.reshape(-1, 1) if your data has a single feature or X.reshape(1, -1) if it contains a single sample.
DeprecationWarning)
/Users/anthonycelio/t2/venv/lib/python3.5/site-packages/sklearn/utils/validation.py:395: DeprecationWarning: Passing 1d arrays as data is deprecated in 0.17 and will raise ValueError in 0.19. Reshape your data either using X.reshape(-1, 1) if your data has a single feature or X.reshape(1, -1) if it contains a single sample.
DeprecationWarning)
today i will None None None None None None None None None None
Two other questions in addition to this... Why does it download the nltk_data punkt tokenizers every time AND Can this be made to write a new sentence without being given the dataset each time? In other words, can I have it read HarryPotter(large).txt just once keep feeding it different fragments to make sentences out without having to fit it with the same data over and over?
(venv) ANTHONYs-iMac:t2 anthonycelio$ python3 main.py -ts "today i will" -tsc 10 -td datasets/harrypotterxxl.txt [nltk_data] Downloading package punkt to [nltk_data] /Users/anthonycelio/nltk_data... [nltk_data] Unzipping tokenizers/punkt.zip. Test sequence generation size changed to: 10 Training data load locaiton changed to: "datasets/harrypotterxxl.txt" Loading text data from: (datasets/harrypotterxxl.txt) Data load successful. WordCount: 15829 Beginning sentence structure parse... Data normalised successful... Beginning sentence vocabulary parse... Data successfully fitted to the sentence structure network. Vectors: 15826 No training data for vocab identifier: SYM No training data for vocab identifier: $ No training data for vocab identifier: POS No training data for vocab identifier: `` No training data for vocab identifier: LS Data successfully fitted to the vocabulary network. Vectors: 15881
/Users/anthonycelio/t2/venv/lib/python3.5/site-packages/sklearn/utils/validation.py:395: DeprecationWarning: Passing 1d arrays as data is deprecated in 0.17 and will raise ValueError in 0.19. Reshape your data either using X.reshape(-1, 1) if your data has a single feature or X.reshape(1, -1) if it contains a single sample. DeprecationWarning) /Users/anthonycelio/t2/venv/lib/python3.5/site-packages/sklearn/utils/validation.py:395: DeprecationWarning: Passing 1d arrays as data is deprecated in 0.17 and will raise ValueError in 0.19. Reshape your data either using X.reshape(-1, 1) if your data has a single feature or X.reshape(1, -1) if it contains a single sample. DeprecationWarning) /Users/anthonycelio/t2/venv/lib/python3.5/site-packages/sklearn/utils/validation.py:395: DeprecationWarning: Passing 1d arrays as data is deprecated in 0.17 and will raise ValueError in 0.19. Reshape your data either using X.reshape(-1, 1) if your data has a single feature or X.reshape(1, -1) if it contains a single sample. DeprecationWarning) /Users/anthonycelio/t2/venv/lib/python3.5/site-packages/sklearn/utils/validation.py:395: DeprecationWarning: Passing 1d arrays as data is deprecated in 0.17 and will raise ValueError in 0.19. Reshape your data either using X.reshape(-1, 1) if your data has a single feature or X.reshape(1, -1) if it contains a single sample. DeprecationWarning) /Users/anthonycelio/t2/venv/lib/python3.5/site-packages/sklearn/utils/validation.py:395: DeprecationWarning: Passing 1d arrays as data is deprecated in 0.17 and will raise ValueError in 0.19. Reshape your data either using X.reshape(-1, 1) if your data has a single feature or X.reshape(1, -1) if it contains a single sample. DeprecationWarning) /Users/anthonycelio/t2/venv/lib/python3.5/site-packages/sklearn/utils/validation.py:395: DeprecationWarning: Passing 1d arrays as data is deprecated in 0.17 and will raise ValueError in 0.19. Reshape your data either using X.reshape(-1, 1) if your data has a single feature or X.reshape(1, -1) if it contains a single sample. DeprecationWarning) /Users/anthonycelio/t2/venv/lib/python3.5/site-packages/sklearn/utils/validation.py:395: DeprecationWarning: Passing 1d arrays as data is deprecated in 0.17 and will raise ValueError in 0.19. Reshape your data either using X.reshape(-1, 1) if your data has a single feature or X.reshape(1, -1) if it contains a single sample. DeprecationWarning) /Users/anthonycelio/t2/venv/lib/python3.5/site-packages/sklearn/utils/validation.py:395: DeprecationWarning: Passing 1d arrays as data is deprecated in 0.17 and will raise ValueError in 0.19. Reshape your data either using X.reshape(-1, 1) if your data has a single feature or X.reshape(1, -1) if it contains a single sample. DeprecationWarning) /Users/anthonycelio/t2/venv/lib/python3.5/site-packages/sklearn/utils/validation.py:395: DeprecationWarning: Passing 1d arrays as data is deprecated in 0.17 and will raise ValueError in 0.19. Reshape your data either using X.reshape(-1, 1) if your data has a single feature or X.reshape(1, -1) if it contains a single sample. DeprecationWarning) /Users/anthonycelio/t2/venv/lib/python3.5/site-packages/sklearn/utils/validation.py:395: DeprecationWarning: Passing 1d arrays as data is deprecated in 0.17 and will raise ValueError in 0.19. Reshape your data either using X.reshape(-1, 1) if your data has a single feature or X.reshape(1, -1) if it contains a single sample. DeprecationWarning) today i will None None None None None None None None None None
(venv) ANTHONYs-iMac:t2 anthonycelio$