[FYI] We disable the "Create New Article" function at present

[shaoyijia]

Thanks everyone for trying our demo and providing feedback!

Due to very high demand, we have to disable the "Create New Article" function at present since our search API has exceeded its maximum limit in less than two days (#12). You can still see examples on the web demo or opt in for email notification when we bring the function back.

Also, as we are researching on automatic content curation, supporting customized sources is on our roadmap. PRs on integrating other sources are welcome!

[allyssonallan]

Congrats for the success! I am looking forward for new alternatives to test it.

[Yucheng-Jiang]

Update: We're gradually rolling out emails to users to opt-in email notification!

[allyssonallan]

Superb @Yucheng-Jiang I tested today! It is really cool!

[HolgerGelhausen]

Thank you very much, I have installed the software on my Mac. The research function works wonderfully. But the Writer gives errors, is the message to be understood as meaning that local writing is no longer possible?

[Yucheng-Jiang]

@HolgerGelhausen would you mind attach your error messages, steps to reproduce?

[HolgerGelhausen]

yes of course:

python -m scripts.run_writing --input-source console --engine gpt-4 --do-polish-article --remove-duplicate Topic: ai_storytelling sentence_transformers.SentenceTransformer : INFO : Load pretrained SentenceTransformer: paraphrase-MiniLM-L6-v2 /opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/transformers/utils/generic.py:311: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead. torch.utils._pytree._register_pytree_node( /opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/transformers/utils/generic.py:311: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead. torch.utils._pytree._register_pytree_node( sentence_transformers.SentenceTransformer : INFO : Use pytorch device: cpu Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in _run_code File "/Users/holgergelhausen/storm-main/src/scripts/run_writing.py", line 94, in main(parser.parse_args()) File "/Users/holgergelhausen/storm-main/src/scripts/run_writing.py", line 54, in main runner.run(topic=topic, File "/Users/holgergelhausen/storm-main/src/engine.py", line 414, in run article = self._generate_article(topic, outline, url_to_info, callback_handler) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/holgergelhausen/storm-main/src/engine.py", line 26, in wrapper result = func(*args, kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/Users/holgergelhausen/storm-main/src/engine.py", line 301, in _generate_article section_result, section_refs = future.result() ^^^^^^^^^^^^^^^ File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/concurrent/futures/_base.py", line 449, in result return self.get_result() ^^^^^^^^^^^^^^^^^^^ File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/concurrent/futures/_base.py", line 401, in get_result raise self._exception File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, *self.kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/holgergelhausen/storm-main/src/engine.py", line 276, in gen_section url_to_snippets = search_collected_info.search(search_qs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/holgergelhausen/storm-main/src/modules/write_page.py", line 220, in search sim = cosine_similarity([encoded_query], self.encoded_snippets)[0] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/sklearn/utils/_param_validation.py", line 214, in wrapper return func(args, kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/sklearn/metrics/pairwise.py", line 1578, in cosine_similarity X, Y = check_pairwise_arrays(X, Y) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/sklearn/metrics/pairwise.py", line 173, in check_pairwise_arrays Y = check_array( ^^^^^^^^^^^^ File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/sklearn/utils/validation.py", line 938, in check_array raise ValueError( ValueError: Expected 2D array, got 1D array instead: array=[]. Reshape your data either using array.reshape(-1, 1) if your data has a single feature or array.reshape(1, -1) if it contains a single sample.

[Yucheng-Jiang]

@HolgerGelhausen the error starts from sim = cosine_similarity([encoded_query], self.encoded_snippets)[0], which suggests that either encoded_query or encoded_snippets is empty. I highly suspect at early stages of pre-writing (research functions), it fails to retrieve any information snippets. To check if this is the case, could you check if observe a bunch of root : ERROR : Error occurs when searching query xxx: 'hits'? If this is the case, it's probably due to You.com api rate limit exceeds.

[HolgerGelhausen]

Thanks for help!!

I did not have the full overview of Storm. I will check the parameters later, this is the folder of the search.

Thanks for helping!!

[Yucheng-Jiang]

@HolgerGelhausen of course! the file raw_search_results.json contains the "research function" results. It should be empty if search engine api (you.com api) could successfully retrieve information. Feel free to follow up in this thread or open new issues if you run into any problem.

[HolgerGelhausen]

Thank you very much, yes it was a typo in the you apikey. I got an article now (after running the pre-stage again), but at the end i got this error message, something to look at? Thanks for help! Great Product! storm_gen_article.txt

engine : INFO : _generate_article executed in 42.2798 seconds Traceback (most recent call last): File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/urllib3/connectionpool.py", line 715, in urlopen httplib_response = self._make_request( ^^^^^^^^^^^^^^^^^^^ File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/urllib3/connectionpool.py", line 467, in _make_request six.raise_from(e, None) File "", line 3, in raise_from File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/urllib3/connectionpool.py", line 462, in _make_request httplib_response = conn.getresponse() ^^^^^^^^^^^^^^^^^^ File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/http/client.py", line 1390, in getresponse response.begin() File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/http/client.py", line 325, in begin version, status, reason = self._read_status() ^^^^^^^^^^^^^^^^^^^ File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/http/client.py", line 294, in _read_status raise RemoteDisconnected("Remote end closed connection without" http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/requests/adapters.py", line 486, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/urllib3/connectionpool.py", line 799, in urlopen retries = retries.increment( ^^^^^^^^^^^^^^^^^^ File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/urllib3/util/retry.py", line 550, in increment raise six.reraise(type(error), error, _stacktrace) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/urllib3/packages/six.py", line 769, in reraise raise value.with_traceback(tb) File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/urllib3/connectionpool.py", line 715, in urlopen httplib_response = self._make_request( ^^^^^^^^^^^^^^^^^^^ File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/urllib3/connectionpool.py", line 467, in _make_request six.raise_from(e, None) File "", line 3, in raise_from File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/urllib3/connectionpool.py", line 462, in _make_request httplib_response = conn.getresponse() ^^^^^^^^^^^^^^^^^^ File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/http/client.py", line 1390, in getresponse response.begin() File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/http/client.py", line 325, in begin version, status, reason = self._read_status() ^^^^^^^^^^^^^^^^^^^ File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/http/client.py", line 294, in _read_status raise RemoteDisconnected("Remote end closed connection without" urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/openai/api_requestor.py", line 606, in request_raw result = _thread_context.session.request( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/requests/adapters.py", line 501, in send raise ConnectionError(err, request=request) requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in _run_code File "/Users/holgergelhausen/storm-main/src/scripts/run_writing.py", line 94, in main(parser.parse_args()) File "/Users/holgergelhausen/storm-main/src/scripts/run_writing.py", line 54, in main runner.run(topic=topic, File "/Users/holgergelhausen/storm-main/src/engine.py", line 419, in run polished_article = self._polish_article(topic, article, remove_duplicate, callback_handler) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/holgergelhausen/storm-main/src/engine.py", line 26, in wrapper result = func(*args, kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/Users/holgergelhausen/storm-main/src/engine.py", line 343, in _polish_article polish_result = polish_page(topic=topic, draft_page=article, polish_whole_page=remove_duplicate) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/dspy/primitives/program.py", line 29, in call return self.forward(*args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/holgergelhausen/storm-main/src/modules/write_page.py", line 295, in forward page = self.polish_page(draft_page=draft_page).page ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/dspy/predict/predict.py", line 60, in call return self.forward(kwargs) ^^^^^^^^^^^^^^^^^^^^^^ File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/dspy/predict/predict.py", line 87, in forward x, C = dsp.generate(signature, config)(x, stage=self.stage) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/dsp/primitives/predict.py", line 78, in do_generate completions: list[dict[str, Any]] = generator(prompt, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/holgergelhausen/storm-main/src/modules/utils.py", line 73, in call response = self.request(prompt, *kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/backoff/_sync.py", line 105, in retry ret = target(args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/dsp/modules/gpt3.py", line 136, in request return self.basic_request(prompt, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/dsp/modules/gpt3.py", line 109, in basic_request response = chat_request(kwargs) ^^^^^^^^^^^^^^^^^^^^^^ File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/dsp/modules/gpt3.py", line 247, in chat_request return _cached_gpt3_turbo_request_v2_wrapped(kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/dsp/modules/cache_utils.py", line 17, in wrapper return func(args, kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/dsp/modules/gpt3.py", line 221, in _cached_gpt3_turbo_request_v2_wrapped return _cached_gpt3_turbo_request_v2(kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/joblib/memory.py", line 655, in call return self._cached_call(args, kwargs)[0] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/joblib/memory.py", line 598, in _cached_call out, metadata = self.call(args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/joblib/memory.py", line 856, in call output = self.func(args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/dsp/modules/gpt3.py", line 216, in _cached_gpt3_turbo_request_v2 return cast(OpenAIObject, openai.ChatCompletion.create(kwargs)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/openai/api_resources/chat_completion.py", line 25, in create return super().create(args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/openai/api_resources/abstract/engine_apiresource.py", line 155, in create response, , api_key = requestor.request( ^^^^^^^^^^^^^^^^^^ File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/openai/api_requestor.py", line 289, in request result = self.request_raw( ^^^^^^^^^^^^^^^^^ File "/opt/homebrew/anaconda3/envs/storm/lib/python3.11/site-packages/openai/api_requestor.py", line 619, in request_raw raise error.APIConnectionError( openai.error.APIConnectionError: Error communicating with OpenAI: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

[HolgerGelhausen]

Thanks for helping @Yucheng-Jiang , i wrote an Blog Post in German about Storm with an example about AI Storytelling!! A Game Changer in Content Creation!

https://www.ki-insights.com/110-der-perfekte-artikel-das-storm-projekt-von-stanford/

[Yucheng-Jiang]

@HolgerGelhausen Here's a thread discussing this error: https://community.openai.com/t/api-aborts-my-connection-without-a-reason-anything-i-can-do/104256/35. It should be something to do with connection with openAI api endpoints.

There's one people in above thread suggesting

Turning off my VPN and it works.

[shaoyijia]

Update: We're gradually rolling out emails to users to opt-in email notification!

Latest Update: Unfortunately, the very high demand exceeds our lab budget, so we have to disable the "Create New Article" function again. 4.5k+ people have tried out this web preview and many left very helpful feedback. Thank you so much!!

We are figuring out the possibility to keep hosting the web preview as many people find it useful. In the meanwhile, we are working on refactoring our codebase and planning the development roadmap with the vision of generalizing STORM into an easy-to-use and easy-to-extend knowledge curation engine. Stay tuned!

[HolgerGelhausen]

@shaoyijia The high demand is a great compliment for your demand, the fact that the budget is exceeded is therefore also understandable. Great work and thank you for taking the work to share it here! Thank you for answering and helping!!