aphonsoar
commented
1 year ago Oi @GuiInfo, realmente isso pode acontecer dependendo do volume de dados do arquivo. Imagino que os arquivos atuais sejam maiores do que os arquivos quando o projeto foi desenvolvido. O que precisaria ser feito aqui eh uma adaptacao do codigo pra particionar os arquivos, e assim nao dar lack de memoria. Veja como foi feito no caso do Simples (linhas 381 a 458), eh basicamente isso que tambem deve ser feito para o estabelecimento (linhas 241 a 318). Isso eh uma melhoria em backlog, no entanto, precisaria de tempo pra desenvolver isso.
italojohnny
Eai Galera, alguem pode me ajudar? Estou tentando realizar o procedimento na tabela estabeleciemento, porém acusa esse erro: OBS - Todas as outras tabelas subiram só essa dá esse erro meu pc tem 32gb de ram e acusa esse erro.
Trabalhando no arquivo: K3241.K03200Y0.D20709.ESTABELE [...] Traceback (most recent call last): File "C:\Users\guilh\AppData\Local\Programs\Python\Python310\lib\site-packages\sqlalchemy\engine\base.py", line 1800, in _execute_context context = constructor( File "C:\Users\guilh\AppData\Local\Programs\Python\Python310\lib\site-packages\sqlalchemy\engine\default.py", line 996, in _init_compiled self.compiled_parameters = [ File "C:\Users\guilh\AppData\Local\Programs\Python\Python310\lib\site-packages\sqlalchemy\engine\default.py", line 997, in
compiled.construct_params(
File "C:\Users\guilh\AppData\Local\Programs\Python\Python310\lib\site-packages\sqlalchemy\sql\compiler.py", line 992, in construct_params
pd[escaped_name] = params[bindparam.key]
MemoryError
The above exception was the direct cause of the following exception:
Traceback (most recent call last): File "C:\Users\guilh\Downloads\projeto\code\ETL_coletar_dados_e_gravar_BD.py", line 173, in
estabelecimento.to_sql(name='estabelecimento', con=engine, if_exists='append', index=False)
File "C:\Users\guilh\AppData\Local\Programs\Python\Python310\lib\site-packages\pandas\core\generic.py", line 2951, in to_sql
return sql.to_sql(
File "C:\Users\guilh\AppData\Local\Programs\Python\Python310\lib\site-packages\pandas\io\sql.py", line 697, in to_sql
return pandas_sql.to_sql(
File "C:\Users\guilh\AppData\Local\Programs\Python\Python310\lib\site-packages\pandas\io\sql.py", line 1739, in to_sql
total_inserted = sql_engine.insert_records(
File "C:\Users\guilh\AppData\Local\Programs\Python\Python310\lib\site-packages\pandas\io\sql.py", line 1332, in insert_records
raise err
File "C:\Users\guilh\AppData\Local\Programs\Python\Python310\lib\site-packages\pandas\io\sql.py", line 1322, in insert_records
return table.insert(chunksize=chunksize, method=method)
File "C:\Users\guilh\AppData\Local\Programs\Python\Python310\lib\site-packages\pandas\io\sql.py", line 950, in insert
num_inserted = exec_insert(conn, keys, chunk_iter)
File "C:\Users\guilh\AppData\Local\Programs\Python\Python310\lib\site-packages\pandas\io\sql.py", line 857, in _execute_insert
result = conn.execute(self.table.insert(), data)
File "C:\Users\guilh\AppData\Local\Programs\Python\Python310\lib\site-packages\sqlalchemy\engine\base.py", line 1380, in execute
return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)
File "C:\Users\guilh\AppData\Local\Programs\Python\Python310\lib\site-packages\sqlalchemy\sql\elements.py", line 333, in _execute_on_connection
return connection._execute_clauseelement(
File "C:\Users\guilh\AppData\Local\Programs\Python\Python310\lib\site-packages\sqlalchemy\engine\base.py", line 1572, in _execute_clauseelement
ret = self._execute_context(
File "C:\Users\guilh\AppData\Local\Programs\Python\Python310\lib\site-packages\sqlalchemy\engine\base.py", line 1806, in _execute_context
self._handle_dbapi_exception(
File "C:\Users\guilh\AppData\Local\Programs\Python\Python310\lib\site-packages\sqlalchemy\engine\base.py", line 2124, in _handle_dbapiexception
util.raise(
File "C:\Users\guilh\AppData\Local\Programs\Python\Python310\lib\site-packages\sqlalchemy\util\compat.py", line 208, in raise_
raise exception
File "C:\Users\guilh\AppData\Local\Programs\Python\Python310\lib\site-packages\sqlalchemy\engine\base.py", line 1800, in _execute_context
context = constructor(
File "C:\Users\guilh\AppData\Local\Programs\Python\Python310\lib\site-packages\sqlalchemy\engine\default.py", line 996, in _init_compiled
self.compiled_parameters = [
File "C:\Users\guilh\AppData\Local\Programs\Python\Python310\lib\site-packages\sqlalchemy\engine\default.py", line 997, in
compiled.construct_params(
File "C:\Users\guilh\AppData\Local\Programs\Python\Python310\lib\site-packages\sqlalchemy\sql\compiler.py", line 992, in construct_params
pd[escaped_name] = params[bindparam.key]